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Executive Summary 


The Regional Educational Laboratories (RELs) are a networked system of 10 organizations that 
serve the educational needs of 10 designated regions across the United States and its territories. The 
U.S. Department of Education (ED) is authorized by the Education Sciences Reform Act (ESRA) to 
award contracts to 10 RELs to support applied research, development, wide dissemination, and 
technical assistance activities. 1 The REL program is administered by the Knowledge Utilization 
Division of the National Center for Education Evaluation and Regional Assistance (NCEERA) 
within ED’s Institute of Education Sciences (TES), which was established by ESRA in 2002. 

ESRA requires NCEERA to provide for independent evaluations of each of the RELs in carrying 
out their duties and to transmit these results to Congress, the National Board for Education 
Sciences, and the appropriate REL governing boards. 2 In 2009, the Evaluation Division of the 
NCEERA contracted with Westat to conduct these evaluations for each REL funded between 2006 
and 2011, as well as an evaluation of the REL program as a whole. 

The interim report from the REL evaluation addresses the following questions: 

• What activities did the RELs undertake to fulfill the missions specified in ESRA? 

• What were the technical quality and relevance of REL Fast Response Project reports 
published by IES and of the corresponding proposals? 

In semi-structured, open-ended interviews with members of the evaluation study team, REL 
staff reported activities under each of the 10 missions of the REL program specified in 
ESRA. Each REL conducted its activities in response to a statement of work (SOW) developed by 
NCEERA in 2005 for the REL contracts in place between 2006 and 2011. The SOW included tasks 
that aligned explicitly with 6 of the 10 statutory missions: (1) provision of training and technical 
assistance, (2) dissemination of scientifically valid research, (3) identification of regional needs, (4) 
performance of applied research projects, (5) provision of educational research in usable forms, and 
(6) collaboration with other ED-funded technical assistance providers. Four additional statutory 
missions — focusing on school finance systems, alternative administrative structures, school 


1 The portion of ESRA pertaining to the RELs, Section 174 of P.L. 107-279, is available at http:/ /ies. ed.gov/ pdf/PLl 07-279.pdf 
[accessed April 19, 2013]. 

2 The evaluation requirement is specified in Section 1 74(j) of ESRA. 
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improvement strategies, and educational technology — were not explicitly in the SOW for the RELs, 
but RELs reported activities under these missions as well. The body of the report documents the 
activities described by staff members from each REL. 

Under the contracts in place between 2006 and 2011, the RELs performed two broad categories of 
applied research projects to prepare reports that were released and disseminated as IES publications. 
Fast response projects (FRPs) were short-term education research and technical assistance projects 
intended to (1) respond to regional and national education needs and priorities and (2) inform policy 
and practice. FRPs used various methods, such as literature reviews, analyses of extant data, and 
qualitative studies. Impact studies were projects designed specifically to make causal inferences about 
an intervention, policy, or practice, typically using randomized controlled trials or regression 
discontinuity designs. The technical quality and relevance of FRP reports and proposals is 
documented in this interim evaluation report, while the technical quality and relevance of impact 
study reports and proposals will be documented in the final evaluation report. 

The average technical quality and relevance ratings for IES-reviewed and IES-published 
FRP reports and for the corresponding initial proposals were between 3.24 and 3.81 on a 5- 
point scale, with 5 being the highest value. Of 297 proposals for FRPs reviewed by NCEERA 
by December 1 , 2009, 46 percent (137) were accepted, and the REL was authorized to proceed with 
the project. Of 166 reports from FRPs reviewed by IES by December 1, 2009, 55 percent (92) were 
accepted for publication as IES reports. Independent panels of technical and content experts rated 
the technical quality and relevance of 91 FRP reports published by IES by December 1, 2009, as 
well as the 75 initial proposals that led to 88 of these reports. 1 The IES-published reports received a 
mean quality rating of 3.81 on a 5-point scale, while the corresponding proposals received a mean 
quality rating of 3.24. Both of these means fell between the categories of “adequate” and “strong” 
quality. The IES-published FRP reports received a mean relevance rating of 3.64 on a 5-point scale, 
while the corresponding proposals received a mean relevance rating of 3.39. Both of these means fell 
between the categories of “adequate” relevance and “relevant.” 

The sections below provide more background on the REL program and on the data collection and 
analysis conducted by the study team to answer the research questions for the interim evaluation 
report. 


3 Of the 92 reports counted as approved for publication by December 1, 2009, one report was not published until January 2010 and 
was not included in an expert panel review. Three of the 91 FRP reports published by IES by December 1, 2009, were without a 
written initial proposal available for panel review. 
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Background on the REL Program 


ED, through the Knowledge Utilization Division in NCEERA within IES, awarded 5-year contracts 
to 10 RELs in FY 2006. Table ES-1 lists the states and territories in each region served by a REL, as 
well as the organizations that held the REL contracts from 2006 to 2011. Annual appropriations for 
the REL program varied over the period of performance of the 2006-201 1 REL contracts. 
Appropriations in FY 2006 and under the FY 2007 continuing resolution equaled $65,470 million 
each year. Over the following 3 years, appropriations rose, to $65,569 million in FY 2008, $67,569 
million in FY 2009, and $70,650 million in FY 2010. For FY 201 1 and FY 2012, appropriations fell 
to $57,535 and $57,426 million, respectively. 


Table ES-1. REL regions and prime contractors, 2006-2011 


Region States and territories Prime contractor 

Appalachia 

Kentucky, Tennessee, Virginia, West Virginia 

CNA 

Central 

Colorado, Kansas, Missouri, Nebraska, North 
Dakota, South Dakota, Wyoming 

Mid-Continent Research for 
Education and Learning (McREL) 

Mid-Atlantic 

Delaware, District of Columbia, Maryland, New 
Jersey, Pennsylvania 

The Pennsylvania State University 
(PSU), with four primary 
subcontractors: Rutgers University, 
ICF International, ANALYTICA, and 
the Metiri Group 

Midwest 

Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, 
Wisconsin 

Learning Point Associates (LPA) 

Northeast & 
Islands 

Connecticut, Maine, Massachusetts, New 
Hampshire, New York, Puerto Rico, Rhode Island, 
Vermont, Virgin Islands 

Education Development Center 
(EDC), with Learning Innovations 
at WestEd and the American 
Institutes for Research (AIR) as 
primary subcontractors 

Northwest 

Alaska, Idaho, Montana, Oregon, Washington 

Education Northwest 

Pacific 

American Samoa, Federated States of 
Micronesia, Guam, Hawaii, Northern Mariana 
Islands, Republic of the Marshall Islands, 
Republic of Palau 

Pacific Resources for Education 
and Learning (PREL) 

Southeast 

Alabama, Florida, Georgia, Mississippi, North 
Carolina, South Carolina 

SERVE Center, University of North 
Carolina at Greensboro 

Southwest 

Arkansas, Louisiana, New Mexico, Oklahoma, 
Texas 

Edvance Research, Inc. 

West 

Arizona, California, Nevada, Utah 

WestEd 


NOTE: Subcontracting arrangements are not shown and may have varied over time, even if the prime contractor remained the same. The 
end date for the 2006-2011 REL contracts was December 31, 2011. 
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What activities did the RELs undertake? 


Section 174(g) of ESRA specifies 10 missions for the RELs. The first five missions relate to the 
core education research and technical assistance work of each REL: (1) the provision of training 
and technical assistance to constituents; (2) the dissemination of scientifically valid and useful 
research, information, reports, and publications; (3) the ongoing identification of educational needs 
within the region; (4) the performance of applied research projects to address regional needs; and (5) 
the provision of research findings in usable forms to promote school improvement, academic 
achievement, and the closing of achievement gaps and to expand the knowledge base related to 
elementary and secondary education and postsecondary access. The sixth mission is to collaborate 
and coordinate services with other ED-funded technical assistance providers. The seventh and 
eighth missions are to gather information on school finance and on alternative administrative 
structures. The ninth mission is to develop school improvement strategies, and the tenth mission is 
to develop innovative technologies in education. 

In 2005 the Knowledge Utilization Division of the NCEERA developed a statement of work 
(SOW) to award contracts to the 10 RELs. This SOW specified six tasks. The first task was for 
regional education needs analysis, training and technical assistance, and fast-response applied research and development 
projects. The second task was for rigorous applied research and development projects , later characterized as 
impact studies. The remaining four tasks were concerned with the coordination and management of 
REL activities: (3) the National Laboratory Network, (4) regional dissemination ; (5) planning, management, 
and reporting, and (6) coordination of the REL network, website, and Intranet. 

While most of the tasks of the REL statement of work aligned explicitly with the missions specified 
for the RELs in ESRA, others did not. The tasks focused on regional needs identification, applied 
research and development projects, regional dissemination, and cross-REL coordination 
corresponded directly with the statutory missions for the REL program. The National Laboratory 
Network (NLN) task and the planning/management/ reporting task did not correspond explicitly 
with any single REL mission specified in ESRA, but could be understood as supporting the 
performance of the other tasks. The NLN, for example, included Internet pages, accessible to the 
public from the IES website, describing each of the RELs and providing downloadable copies of 
IES-approved REL IES-published reports. The NLN also included an Intranet for internal use in 
cross-REL collaboration and working groups. Four of the statutory missions of the REL program 
specified in ESRA — those focused on school finance, alternative administrative structures, school 
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improvement strategies, and innovative technologies in education — were not explicit in the tasks of 
the SOW. 

To describe what activities the RELs had completed under the 10 missions specified in ESRA, 
including those not specified explicitly in the SOW, the evaluation study team conducted in-person 
interviews with staff members from each REL, including each REL’s director. 4 Interviews with REL 
staff took place between May 2010 and July 2010 and lasted approximately 4 hours per REL. 
Interviewers from the study team asked the RELs to describe up to three major activities under each 
mission. REL staff were permitted to report the same activity as a major activity for more than one 
mission. The study team reviewed transcripts of the interviews to prepare a document that described 
the major REL activities reported by REL staff as addressing each of the 10 missions. To verify the 
accuracy of the write-ups, each REL director reviewed the draft document and made any necessary 
factual corrections. In general, the RELs reported activities under each of the statutory missions for 
the program. 

Other than documenting REL reports published by IES, the evaluation study team made no attempt 
to verify independently the reported information on REL activities, so the accuracy and 
completeness of this information is unknown. Due to the semi-structured and open-ended nature of 
the interviews with REL staff, the information gathered from these interviews is neither a systematic 
nor a comprehensive inventory of all REL program activities conducted between 2006 and 2010. 

The suggested prompts in the interview protocol were not always utilized, so it is possible that some 
activities reported voluntarily by one REL director were not reported by another REL director, even 
if similar activities had occurred for both RELs. Because the interview protocol left the definition of 
a “project” or “major activity” up to the judgment of the REL staff being interviewed, reported 
REL activities are not necessarily comparable across RELs. For these reasons, the documentation 
of REL activities as reported by REL staff is presented only in the REL-specific chapters of this 
report (chapters 4 through 13), and results are not summarized across RELs. 

What were the technical quality and relevance of REL Fast Response Project 
reports published by IES and of the corresponding proposals? 

The study team’s plan to evaluate the quality and relevance of REL Fast Response Project (FRP) 
proposals and reports included (1) gathering data from the Knowledge Utilization Division of the 


4 The protocol developed by the study team for the REL interviews is included in Appendix C. 
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NCEERA on the percentage of FRP proposals accepted for continuation as IES-supported studies, 
(2) gathering data from the Knowledge Utilization Division on the percentage of FRP reports 
accepted for publication as IES reports, and (3) having an independent expert panel review the 
technical quality and relevance of FRP reports released by IES and of the proposals that had 
resulted in those reports. 

Section 186(c) of ESRA requires all research, statistics, and evaluation reports conducted by, or 
supported through, IES to “be subjected to rigorous peer review before being published or 
otherwise made available to the public.” In the case of the REL FRP reports, this review occurred 
through an Analytical and Technical Support (ATS) contract, which NCEERA awarded to 
Mathematica Policy Research, Inc., on September 29, 2006. A REL would first submit a proposal 
for the FRP to its NCEERA project officer for review. Additional review of the proposal occurred 
through anonymous external reviewers working under the ATS contract. These reviews informed 
NCEERA’s decision to reject a proposal, approve the project, or request the REL to revise its 
proposal. For approved projects, RELs would perform the work and submit draft reports to the 
NCEERA project officer for review. External review through the ATS would again inform 
NCEERA’s decision to reject a report or request revisions. The final decision of whether to approve 
an FRP report for publication by IES was made by the NCEERA commissioner based on prior 
review by NCEERA staff and the ATS contractor. The ATS contract ended on March 28, 2012. 

For the calculation of the percentage of FRP proposals and reports approved, the study team 
excluded “proposal or report reviews in process.” The denominator for the percentage of accepted 
FRP proposals includes all of the FRP proposals that, by December 1, 2009, were either accepted 
and authorized to proceed, or else rejected, but excludes proposals for which the NCEERA review 
was in process. The denominator for the percentage of accepted FRP reports includes all of the 
reports that, by December 1, 2009, were either approved for publication or rejected, but excludes 
reports for which the IES review was in process. Readers should note that, even if a REL’s 
proposal for an FRP was accepted by NCEERA and the project was authorized to proceed, the 
resulting report could be rejected by NCEERA as a result of subsequent review by NCEERA or 
ATS contract staff. 

Of 297 proposals for FRPs reviewed by NCEERA by December 1, 2009, 46 percent (137) were 
accepted, and the REL was authorized to proceed with the project. The number of proposals 
submitted by each REL ranged from 17 to 45, and the percentage accepted by IES for each REL 
ranged from 24 to 67 percent. Of 166 FRP reports reviewed by IES by December 1, 2009, 55 
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percent (92) were accepted for publication as IES reports. The number of reports submitted by each 
REL varied from 10 to 27, and the percentage accepted ranged from 25 to 80 percent. The 
percentages of proposals and reports accepted for each REL are provided in the REL-specific 
chapters of the report. 

In addition to calculating acceptance rates for FRP proposals and reports, the study team selected 
IES-published FRP reports and corresponding proposals for independent review by expert panels of 
methodological and content experts. Because of resource limitations for the evaluation, the 
evaluation study team and NCEERA decided to focus panel reviews on the beginning and end of 
IES-published FRP reports. Revised proposals, proposals that were rejected or were still under 
review, and proposals for reports that were not published by IES were all excluded from panel 
review. Consequently, findings on the quality and relevance of proposals refer only to proposals 
resulting in IES-published FRP reports that were released by December 1, 2009. The study team 
collected from REL web sites and from the REL program office all of the FRP reports that met 
these criteria, as well as the corresponding initial proposals that were submitted by the RELs. In 
total, the RELs produced 91 IES-published reports by this date, 88 of which were derived from 75 
initial proposals. In some cases, a single proposal led to multiple IES-published reports (e.g., one for 
each state in a region). 

The study team developed two rubrics for use in the expert panel review: one for reports and one 
for initial proposals (Appendix A). The mbrics included two dimensions: technical quality and 
relevance. Each dimension was further defined by multiple indicators. The rubrics for rating 
proposals and reports shared six indicators of quality and three indicators of relevance. Two 
additional indicators of quality and one additional indicator of relevance were included in the rubric 
for rating reports. Indicators for the dimensions were rated on a 5-point scale, with 1 being the 
lowest rating and 5 being the highest rating (Figure ES-1). 

The study team recruited 48 experts in content and/ or methodology, screened them for conflicts of 
interest, and trained them to use the rubrics for evaluating the quality and relevance of IES- 
published FRP reports and corresponding proposals. Training of panelists occurred in January and 
February 2010. Comments from panelists during the training were used to make minor revisions to 
each rubric. A total of 79 panels were organized to review FRPs, including 46 experts and an average 
of 2.92 experts per panel. The number of panels on which any one expert served averaged 5.02 and 
ranged from 2 to 1 0. Three panels reviewed no proposal, since no written initial proposal for the 
FRP was available. All other panels reviewed both the proposal for the FRP and the resulting IES- 
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published report. Five panels reviewed multiple IES-published reports that arose from the same 
proposal. While three experts were generally assigned to review each set of FRP documents based 
on content or methodology expertise, in five cases, fewer experts were assigned to review a product 
because no other panelists had the necessary expertise. Ratings for each FRP document (proposal or 
report) were generated by averaging the scores across all panelists and across dimension-specific 
indicators. Panel reviews and reconciliation phone calls occurred between February and May 2010. 

On average, the expert panels rated IES-published FRP reports and corresponding proposals as 
being between “adequate” and “strong” in quality. The IES-published reports received a mean 
quality rating of 3.81 on a 5-point scale, while the corresponding FRP proposals received a mean 
quality rating of 3.24. For proposals, 9.1 percent of quality ratings were at the highest level of quality 
(“very strong”), while for IES-published reports, 26.9 percent of quality ratings were “very strong” 
(Figure ES-1). 

The expert panels rated IES-published FRP reports and corresponding proposals, on average, as 
being between “adequate” in relevance and “relevant.” The IES-published reports received a mean 
relevance rating of 3.64 on a 5-point scale, while the corresponding FRP proposals received a mean 
relevance rating of 3.39. For proposals, 13.0 percent of relevance ratings were at the highest level of 
relevance (“highly relevant”), while for IES-published reports, 19.3 percent of relevance ratings were 
at the level of “highly relevant” (Figure ES-1). 

The body of the report provides more detailed REL-specific findings on the quality and relevance of 
the FRP reports and corresponding proposals. 
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Figure ES-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from all 10 
RELs combined 
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Figure Reads: 9.1 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on a 
5-point scale. 

NOTE: The distributions for proposal quality and relevance were based on 1,280 and 656 indicator-level ratings, respectively, which is 
largely attributed to differences in number of indicators associated with quality and relevance. The distributions for lES-published report 
quality and relevance were based on 2,051 and 1,065 indicator-level ratings, respectively. The difference in number of indicators 
associated with quality and relevance between lES-published reports and proposals is due to the fact the number of lES-published reports 
is larger than that of proposals (see page xv). Ratings of “not applicable” were not included in these frequency distributions. 
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The Regional Educational Laboratories (RELs) are a networked system of 10 organizations that serve 
the educational needs of 10 designated regions across the United States and its territories (Table 1-1). 
The U.S. Department of Education (ED) is authorized by the Education Sciences Reform Act (ESRA) 
to award contracts to 10 RELs to support applied research, development, wide dissemination, and 
technical assistance (TA) activities. ' The REL program is administered by the Knowledge Utilization 
Division of the National Center for Education Evaluation and Regional Assistance (NCEERA) within 
ED’s Institute of Education Sciences (IES), which was established by ESRA in 2002. 


ED, through the Knowledge Utilization Division of the NCEERA within IES, awarded 5-year 
contracts to 10 RELs on a competitive basis in FY 2006. These contracts were subsequently extended 
to end in FY 2012. Table 1-2 lists the organizations holding the 10 REL contracts from 2006 to 201 1 
as well as their history of REL funding (i.e., whether they have held REL grants/ contracts in the past). 

Table 1-1. States and territories served by each REL 


Region 


States and territories 


Appalachia 

Central 

Mid-Atlantic 

Midwest 

Northeast & Islands 

Northwest 

Pacific 

Southeast 

Southwest 

West 


Kentucky, Tennessee, Virginia, West Virginia 

Colorado, Kansas, Missouri, Nebraska, North Dakota, South Dakota, Wyoming 

Delaware, District of Columbia, Maryland, New Jersey, Pennsylvania 

Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, Wisconsin 

Connecticut, Maine, Massachusetts, New Hampshire, New York, Puerto Rico, Rhode 
Island, Vermont, Virgin Islands 

Alaska, Idaho, Montana, Oregon, Washington 

American Samoa, Federated States of Micronesia, Guam, Hawaii, Northern Mariana 
Islands, Republic of the Marshall Islands, Republic of Palau 

Alabama, Florida, Georgia, Mississippi, North Carolina, South Carolina 

Arkansas, Louisiana, New Mexico, Oklahoma, Texas 

Arizona, California, Nevada, Utah 


5 The portion of ESRA pertaining to the Regional Educational Laboratories, Section 174 of P.L. 107-279, is available at http:/ /ies. ed.gov/pdf/PLl 07- 
279.pdf [accessed April 19, 2013]. 
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Table 1-2. 


REL prime contractors, 2006-2011 


Region 

Prime contractor 

History of Funding 

Appalachia 

CNA 

CNA received a REL contract for the first time 
in FY 2006 (awarded February 6, 2006). CNA 
was also awarded a REL contract for this 
region in FY 2012. 

Central 

Mid-Continent Research for Education and 
Learning (McREL) 

McREL had held the REL grant/contract 
continuously since 1966. The FY 2006 contract 
was awarded on January 20, 2006. 

Mid-Atlantic 

The Pennsylvania State University (PSU), with 4 
primary subcontractors: Rutgers University, ICF 
International, ANALYTICA, and the Metiri Group 

PSU received a REL contract for the first time 
in FY 2006 (awarded March 23, 2006). 

Midwest 

Learning Point Associates (LPA) 

LPA had held a REL grant/contract since 1984. 
The FY 2006 contract was awarded on March 
9, 2006. The American Institutes for Research, 
(AIR) which merged with LPA in 2011, was 
awarded a REL contract for this region in FY 
2012. 

Northeast & Islands 

Education Development Center (EDC), with 
Learning Innovations at WestEd and AIR as 
primary subcontractors 

EDC held one of the original REL grants but did 
not hold one immediately before the FY 2006 
award. The FY 2006 contract was awarded on 
March 15, 2006. EDC was also awarded a REL 
contract for this region in FY 2012. 

Northwest 

Education Northwest 

Education Northwest, previously known as 
Northwest Regional Educational Laboratory, 
had held the REL grant/contract since 1966. 
The FY 2006 contract was awarded on 
February 1, 2006. Education Northwest was 
also awarded a REL contract for this region in 
FY 2012. 

Pacific 

Pacific Resources for Education and Learning 
(PREL) 

PREL had held the REL grant/contract since 
1990. The FY 2006 contract was awarded on 
March 16, 2006. 

Southeast 

SERVE Center, University of North Carolina at 
Greensboro 

SERVE had held the REL grant/contract since 
1990. The FY 2006 contract was awarded on 
March 16, 2006. 

Southwest 

Edvance Research, Inc. 

Edvance received a REL contract for the first 
time in FY 2006 (awarded March 15, 2006). 

West 

WestEd 

WestEd had held the REL grant/contract since 
1966. The FY 2006 contract was awarded on 
January 18, 2006. WestEd was also awarded a 
REL contract for this region in FY 2012. 

NOTE: Subcontracting arrangements are not shown and may have varied over time, even if the prime contractor remained the same. The 
end date for the 2006-2011 REL contracts was December 31, 2011. 

The table shows that three contractors (CNA, Pennsylvania State University, and Edvance Research, 
Inc.) held first-time REL contracts in FY 2006; four contractors (Learning Point Associates, 
Education Development Center, Pacific Resources for Education and Learning, and SERVE Center 
at the University of North Carolina at Greensboro) held previous REL contracts; and three 
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contractors (Mid-Continent Research for Education and Learning, Education Northwest, and 
WestEd) held continuous REL contracts since the inception of the program in 1966. The table notes 
whether the same prime contractor was awarded a REL contract for the same region in FY 2012. 

Annual appropriations for the REL program varied over the period of performance of the 2006- 
2011 REL contracts. Appropriations in FY 2006 and under the FY 2007 continuing resolution 
equaled $65,470 million each year. Over the following 3 years, appropriations rose, to $65,569 
million in FY 2008, $67,569 million in FY 2009, and $70,650 million in FY 2010. For FY 201 1 and 
FY 2012, appropriations fell to $57,535 and $57,426 million, respectively. 

Missions of the REL Program 


While the REL program was begun in 1966, it was most recently reauthorized under ESRA in 2002. 

Section 174(g) of ESRA specifies 10 missions for the RELs: 

1. Provide training and technical assistance to constituents 

2. Disseminate scientifically valid research, information, reports, and publications that are 
usable for improving academic achievement, closing achievement gaps, and encouraging 
and sustaining school improvement 

3. Develop a plan for identifying and serving the needs of the region by conducting a 
continuing survey of the educational needs, strengths, and weaknesses within the region 

4. Carry out applied research projects that are designed to serve the particular educational 
needs of the region, that reflect findings from scientifically valid research, and that result 
in user-friendly, replicable school-based classroom applications geared toward promoting 
student achievement 

5. Provide educational applied research in usable forms that promote school-improvement, 
academic achievement, and the closing of the achievement gaps and contribute to the 
current base of education knowledge by addressing problems in elementary and 
secondary education and access to postsecondary education 

6. Collaborate and coordinate services with other technical assistance providers funded by 
ED 

7. Assist in gathering information on school finance systems to promote improved access to 
educational opportunities and to better serve all public school students 
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8. Assist in gathering information on alternative administrative structures that are more 
conducive to planning, implementing, and sustaining school reform and improved 
academic achievement 

9. Bring teams of experts together to develop and implement school improvement plans 
and strategies, especially in low-performing or high-poverty schools 

10. Develop innovative approaches to the application of technology in education that are 
unlikely to originate from within the private sector, but which could result in the 
development of new forms of education software, education content, and technology- 
enabled pedagogy 

In 2005, the Knowledge Utilization Division of the NCEERA developed a statement of work 
(SOW) to award contracts to the 10 RELs. This SOW specified six tasks: 

• Task 1: Regional education needs analysis, training and technical assistance, and fast- 
response applied research and development projects; 

• Task 2: Rigorous applied research and development; 

• Task 3: National Laboratory Network (NLN); 

• Task 4: Regional dissemination; 

• Task 5: Planning, management, and reporting; and 

• Task 6: Coordination of REL network, website, and Intranet. 

While most of the tasks of the REL statement of work aligned explicitly with the missions specified 
for the RELs in ESRA, others did not. The tasks focused on regional needs identification (Task 1), 
applied research and development projects (Tasks 1 and 2), regional dissemination (Task 4), and 
cross-REL coordination (Task 6) corresponded directly with the first six statutory missions for the 
REL program (Table 1-3). The National Laboratory Network (NLN) task (Task 3) and the 
planning/ management/ reporting task (Task 5) did not correspond explicitly with any single REL 
mission specified in ESRA, but could be understood as supporting the performance of the other 
tasks. The NLN, for example, included Internet pages, accessible to the public from the IES 
website, describing each of the RELs and providing downloadable copies of IES-approved REL 
IES-published reports. The NLN also included an Intranet for internal use in cross-REL 
collaboration and working groups. The last four statutory missions of the REL program specified in 
ESRA — those focused on school finance, alternative administrative structures, school improvement 
strategies, and innovative technologies in education — were not explicit in the tasks of the SOW. 
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Table 1-3. 


Alignment of REL statement of work tasks with statutory missions for the RELs 


ESRA mission 


Task in the REL Statement of Work 


1. Provide training and technical assistance to constituents 1. Regional education needs analysis, training 

and technical assistance, and fast-response 
applied research and development projects 


2. Disseminate scientifically valid research, information, reports, and 4. Regional dissemination 
publications that are usable for improving academic achievement, 
closing achievement gaps, and encouraging and sustaining school 
improvement 


3. Develop a plan for identifying and serving the needs of the region 
by conducting a continuing survey of the educational needs, 
strengths, and weaknesses within the region 


1. Regional education needs analysis, training 
and assistance, and fast-response applied 
research and development projects 


4. Carry out applied research projects that are designed to serve the 
particular educational needs of the region, that reflect findings 
from scientifically valid research, and that result in user-friendly, 
replicable school-based classroom applications geared toward 
promoting student achievement 

5. Provide educational applied research in usable forms that 
promote school-improvement, academic achievement, and the 
closing of the achievement gaps and contribute to the current 
base of education knowledge by addressing problems in 
elementary and secondary education and access to 
postsecondary education 

6. Collaborate and coordinate services with other technical 
assistance providers funded by the Department of Education 

7. Assist in gathering information on school finance systems to 
promote improved access to educational opportunities and to 
better serve all public school students 

8. Assist in gathering information on alternative administrative 
structures that are more conducive to planning, implementing, 
and sustaining school reform and improved academic 
achievement 

9. Bring teams of experts together to develop and implement school 
improvement plans and strategies, especially in low-performing or 
high-poverty schools 

10. Develop innovative approaches to the application of technology in 
education that are unlikely to originate from within the private 
sector, but which could result in the development of new forms of 
education software, education content, and technology-enabled 
pedagogy 

Not explicitly included in the REL missions 
Not explicitly included in the REL missions 


1. Regional education needs analysis, training 
and assistance, and fast-response applied 
research and development projects 


2. Rigorous applied research and development 


6. Coordination of REL network, website, and 
Intranet 

Not explicitly included in the REL statement of 
work 


Not explicitly included in the REL statement of 
work 


Not explicitly included in the REL statement of 
work 


Not explicitly included in the REL statement of 
work 


3. National Laboratory Network 
5. Planning, management, and reporting 


Table Reads: The first statement of work task, Regional education needs analysis, training and technical assistance, and fast-response 
applied research and development projects, aligned with the first mission, provide training and technical assistance to constituents. 

SOURCE: Education Sciences Reform Act (P.L. 107-279) and the statement of work developed by NCEERA in 2005. 
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Under the contracts in place between 2006 and 2011, the RELs performed two broad categories of 
projects to prepare reports that were released and disseminated as IES publications. Fast response 
projects (FRPs) were short-term education research and technical assistance projects intended to (1) 
respond to regional and national education needs and priorities and (2) inform policy and practice. 
FRPs used various methods, such as literature reviews, analyses of extant data, and qualitative 
studies. Impact studies were projects designed specifically to make causal inferences about an 
intervention, policy, or practice, typically using randomized controlled trials or regression 
discontinuity designs. To assist the reader in understanding the content of the remainder of the 
report, Table 1-4 provides definitions and explanations of these and other terms describing the work 
of the REL program between 2006 and 2011. 

Independent Evaluation of the RELs 

Section 174(j) of ESRA requires NCEERA to provide for independent evaluations of each of the 
RELs in carrying out their duties, and transmit these results to Congress, the National Board for 
Education Sciences, and the appropriate REL governing boards. In 2009, the Evaluation Division of 
the NCEERA — which is administratively distinct from the Knowledge Utilization Division that 
manages the REL program — contracted with Westat to conduct these evaluations as well as an 
evaluation of the REL program as a whole. The REL program evaluation was designed to address, 
for the program and for each REL funded between 2006 and 2011, the following questions: 

• What activities did the RELs undertake to fulfill the missions specified in ESRA? 

• What were the technical quality and relevance of REL Fast Response Project reports 
published by IES and of the corresponding proposals? 

• What were the technical quality and relevance of the REL impact study reports published by 
IES and of the corresponding proposals? 

• How relevant and useful were the REL technical assistance products to the needs of the 
states, localities, and policymakers in their regions? 

This interim evaluation report addresses the first two research questions for the evaluation; the other 
two questions will be addressed in the final evaluation report. 
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Table 1-4. 


Definition and explanation of terms describing the work of the RELs, 2006-2011 


Activity 

Description 

Fast Response Projects 
(FRPs) 

Short-term projects intended to inform policy and practice, FRPs used various methods— 
such as literature reviews, analysis of existing data sets, and qualitative studies— to inform 
decision-making. Designed to be completed within 1 year, FRPs could report on recent data, 
studies, or reviews or tap into state or district resources to summarize the evidence on 
educational issues or problems or on educational conditions or trends. They could also 
summarize scientific evidence through descriptive analyses of local, regional, or national 
data or of existing studies or research summaries. REL staff and/or subcontractors designed 
FRPs and collected data for these studies. 

Bridge Events, formerly 
called Experts-Bring- 
Evidence-to-Practitioners 

Interactive forums conducted by the RELs that brought together lES-approved experts with 
education practitioners and policymakers. These events could be held in-person or via 
webinars. 

Collaboration/ 

coordination 

ED awarded REL Mid-Atlantic responsibilities for specific coordination tasks (based on a 
competitive bid process among the RELs). These activities included support for the NLN 
Internet, NLN Intranet, six cross-REL working groups, REL Reference Desk, Ask-A-REL, and 
Thursdays @ 3. Although REL Mid-Atlantic was responsible for coordinating certain REL 
tasks, all RELs participated in the collaboration. 

National Laboratory 
Network (NLN) Internet 

Housed on the IES website, the NLN Internet was for public use. it contained web pages for 
each of the RELs and provides links to REL report publications. Alternatively, the NLN 
Intranet was also for internal REL use. It was primarily used for cross-REL collaboration and 
working group space. 

Working groups 

A form of REL collaboration that included working groups on: (1) data, (2) dissemination, (3) 
rural issues, (4) the American Recovery and Reinvestment Act (ARRA), (5) the REL Reference 
Desk, and (6) the NLN. Working groups typically met by phone on either a regular schedule or 
intermittently. 

REL Reference Desk 

In collaboration with the National Library of Education, the REL Reference Desk responded to 
technical assistance questions and listed experts available to answer technical questions 
that have not been previously asked and responded to by the RELs. All labs could use the 
Reference Desk to respond to their individual technical assistance requests. 

Ask-A-REL 

A component on each individual REL’s webpage that allowed users to ask questions of 
specific RELs and receive corresponding responses from REL staff. Responses were 
generally provided by email, or through the IES website or the REL’s own website. 

Thursdays @ 3 

Bimonthly teleconference to promote discussion and collaboration among the RELs. 
Project officers from the Knowledge Utilization Division of the NCEERA usually joined the 
teleconference. 


NOTE: Information for this table was taken from interviews with REL directors as well as a review of REL web sites. 

Report Organization 

The remainder of this report is organized in the following manner. Chapter 2 describes the design of 
the evaluation and specific methods used to address the research questions for the interim 
evaluation report. Data on the FRP proposals and reports accepted by IES, and expert panel reviews 


Evaluation of the Regional Educational Laboratories: Interim Report 

7 


of the quality and relevance of those products, are presented in chapter 3. Ten subsequent chapters 
describe the activities undertaken by each REL and the quality and relevance of each REL’s FRP 
reports and corresponding proposals. 6 Appendix A includes the rubric used by the expert panel, and 
Appendix B describes the process used for assessing inter-rater agreement among expert panel 
members. Appendix C includes the interview protocol used with REL directors. Appendix D lists 
the Technical Working Group (TWG) members advising the evaluation study team. 


6 Due to the open-ended nature of the interviews with REL staff, the documentation of REL activities as reported in those interviews are presented 
only in the REL-specifk chapters of this report, and results are not summarized across RELs. 
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Evaluation Design and Methods 



This report addresses the following two research questions: 

• What activities did the RELs undertake to fulfill the missions specified in ESRA? 

• What were the technical quality and relevance of the REL Fast Response Project reports 
published by IES and of the corresponding proposals? 

The study team for the evaluation used a variety of data collection and analysis activities to address 
these questions. Plans for these activities were developed in consultation with members of a 
Technical Working Group with expertise in program evaluation and education-related technical 
assistance (Appendix D), and with technical guidance from the project officer and leadership within 
the Evaluation Division of the NCEERA. 

To describe the activities the RELs had completed under the 10 REL missions specified in ESRA, 
members of the evaluation study team conducted interviews with staff members from each REL, 
including each REL’s director. To evaluate the technical quality and relevance of FRP reports and 
proposals, the study team obtained information from the REL program office in the NCEERA 
Knowledge Utilization Division on the number of FRP proposals submitted for NCEERA 
approval, and the number of FRP reports approved for release as IES publications. The study team 
also organized expert panels consisting of individuals with relevant content and/ or methodological 
expertise to rate the technical quality and relevance of FRP reports published by IES by December 
1, 2009, and of the corresponding initial proposals. 

Interviews with REL Staff 


To describe the projects and activities the RELs had completed under the 10 missions specified in 
ESRA, including those not specified explicitly in the statement of work for the REL contracts, the 
evaluation study team from Westat and its subcontractor, Policy Studies Associates (PSA), 
conducted in-person interviews with staff members from each REL, including each REL’s director. 
Interviews took place between May 2010 and July 2010 and lasted approximately 4 hours per REL. 
In some cases, the director was accompanied by additional members of the REL’s staff, such as 
those in charge of business, dissemination, research, coordination, and/or the Reference Desk. 
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The evaluation study team used an open-ended interview protocol (Appendix C) that asked REL 
staff to describe up to three major projects or activities their REL completed under each ESRA 
mission. REL staff were told before the interview that they could indicate that no activities had 
been conducted under a mission if that were the case. REL staff were permitted to report the same 
activity as a major activity for more than one mission, and to define “major activity” as they saw fit. 
If REL staff interpreted non-IES published documents 7 or reports developed using REL funds as 
major activities, they were able to report them. Embedded in each overarching question were 
multiple probing questions to ensure that the evaluation study team received detailed and 
comprehensive responses from REL staff. Follow-up questions addressed RELs’ participation in 
collaboration/ coordination activities, such as Thursdays @3, working groups, and Ask-A-REL. The 
interviewers did not invoke the prompts in the protocol in all instances, leaving some responses to 
the judgment of the REL staff being interviewed. 

The interviews with REL staff were taped and then transcribed. Analysts on the study team 
reviewed the transcripts and their interview notes to prepare a document that described the major 
REL activities reported by REL staff as addressing each of the 10 missions. If necessary, additional 
information, such as the complete title for an impact study mentioned in the interview, was taken 
from the REL websites. Since the design of the interview protocol required that REL staff members 
respond to individual questions pertaining to each mission, the study team used the protocol to 
organize responses by mission. To verify the accuracy of the write-ups, each REL director reviewed 
the document and made any necessary factual corrections. 

Limitations of Interviews With REL Staff 


With respect to the results of the interviews with REL directors, it is important to recognize that 
these descriptive findings are based on activities reported by REL staff. Other than documenting 
REL reports published by IES, the evaluation study team made no attempt to verify independently 
the reported information on REL activities, so the accuracy and completeness of this information is 
unknown. Due to the semi-structured and open-ended nature of the interviews with REL staff, the 
information gathered from these interviews is neither a systematic nor a comprehensive reporting of 
all REL program activities conducted between 2006 and 2010. The suggested prompts in the 
interview protocol were not always used, so it is possible that some research or technical assistance 


7 A “non-IES published report” could either be (1) a report disseminated by means other than publication through IES (for example, a report 
published by the organization holding the REL contract), or (2) a report reviewed in draft form by IES but not released to the public. Neither 
NCEERA nor the evaluation study team gathered systematic information on the number of reports in either category. 
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activities reported voluntarily by one REL director were not reported by another REL director, even 
if similar activities had occurred for both RELs. Because the interview protocol left the definition of 
a “project” or “activity” up to the judgment of the REL staff being interviewed, reported REL 
activities are not necessarily comparable across RELs. For these reasons, the documentation of 
REL activities as reported by REL staff is presented only in the REL-specific chapters of this report, 
and results are not summarized across RELs. Chapters 4 through 13 of this report present, by REL 
and for each mission specified in ESRA, the specific research and technical assistance activities 
described by REL directors in their interviews with members of the evaluation study team. 

IES Acceptance of Fast Response Project Proposals and Reports 

The evaluation study team’s plan to assess the quality and relevance of REL FRPs included 
gathering data from the Knowledge Utilization Division of the NCEERA on the percentage of FRP 
proposals accepted for continuation as IES-supported studies, and gathering data from the 
Knowledge Utilization Division on the percentage of FRP reports accepted for publication as IES 
reports. 

Section 186(c) of ESRA requires all research, statistics, and evaluation reports conducted by, or 
supported through, IES, to “be subjected to rigorous peer review before being published or 
otherwise made available to the public.” In the case of REL FRP reports, this review occurred 
through an Analytical and Technical Support (ATS) contract, which NCEERA awarded to 
Mathematica Policy Research, Inc., on September 29, 2006. A REL would first submit a proposal 
for the FRP to its NCEERA project officer for review. Additional review of the proposal occurred 
through anonymous external reviewers working under the ATS contract. These reviews informed 
NCEERA’s decision to reject a proposal, approve the project, or request the REL to revise its 
proposal. For approved projects, RELs would perform the work and submit draft reports to the 
NCEERA project officer for review. External review through the ATS would again inform 
NCEERA’s decision to reject a report or request revisions. The final decision of whether to approve 
an FRP report for publication by IES was made by the NCEERA commissioner based on prior 
review by NCEERA staff and the ATS contractor. The ATS contract ended on March 28, 2012. 

For the calculation of the percentage of FRP proposals approved by IES, the evaluation study team 
excluded proposals being developed by the RELs that had yet to be submitted to the NCEERA. 

The numerator for the percentage of FRP proposals accepted by IES by December 1, 2009, was 
based on the number of all FRP proposals that, by that date, had been accepted by IES and 
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authorized to proceed as a project, regardless of whether a report from the FRP had already been 
approved for IES publication, had been rejected for IES publication, was still under IES review, or 
had not yet been submitted for IES review. The denominator included the total number of FRP 
proposals received for which the status of the proposal had been resolved by December 1, 2009, and 
therefore excluded proposals submitted to the NCEERA for which the proposal review was still in 
process: 


Acceptance rate for FRP proposals = 

# FRP proposals accepted by NCEERA / 

(# FRP proposals received by NCEERA- # FRP proposals with NCEERA review in process) 

For the calculation of the percentage of FRP reports approved as IES publications, the study team 
excluded reports under preparation by the RELs that had yet to be submitted to IES. The 
numerator for the percentage of FRP reports accepted for IES publication by December 1, 2009, 
was equal to the number of reports accepted for IES publication by that date. The denominator 
included the total number reports submitted to IES for which the status of the report had been 
resolved (either acceptance or rejection for IES publication), and therefore excluded reports 
submitted to IES for which IES review was still in process: 

Acceptance rate for reports = 

# FRP reports accepted for IES publication / 

(# FRP reports received by IES- # FRP reports with IES review in process) 

Readers should note that, even if a REL’s proposal for an FRP was accepted by NCEERA and the 
project was authorized to proceed, the resulting report could be rejected by NCEERA as a result of 
subsequent review by NCEERA or ATS contract staff. 

Expert Panel Review of FRPs 

In addition to calculating acceptance rates for FRP proposals and reports, the evaluation study team 
selected FRP reports published by IES and corresponding proposals for independent review by 
expert panels of methodological and content experts. RELs submitted initial as well as revised 
proposals as part of the FRP process. The evaluation study team chose to focus the proposal review 
on initial proposals. While later proposals could be seen as representing the collective work of the 
REL, IES, and its ATS contractor, the initial proposals could be more completely attributed to the 


Evaluation of the Regional Educational Laboratories: Interim Report 

12 


RELs. In addition, by evaluating the initial proposals and IES-published reports, the study is able to 
document ratings before and after the contributions of the IES review process. Because of resource 
limitations for the evaluation, the evaluation study team and NCEE decided to focus panel reviews 
on the beginning and end of IES-published FRP reports. Revised proposals, proposals that were 
rejected or were still under review, and proposals for reports that were not published by IES were all 
excluded from panel review. Consequently, findings on the quality and relevance of proposals refer 
only to proposals resulting in IES-published FRP reports that were released by December 1, 2009. 

The evaluation study team collected from REL web sites and from the REL program office all of 
the accepted FRP reports published by IES on its website, as well as the corresponding initial 
proposals that were produced by the 10 RELs between January 18, 2006 and December 1, 2009. 8 A 
total of 91 FRP reports were published by IES over this period of time. Of these 91 reports, 88 
arose from 75 proposals, out of a total of 297 FRP proposals from the RELs documented over this 
period. Several of the 75 proposals resulted in multiple IES-published FRP reports: two proposals 
each led to five reports, one proposal led to three reports, and three proposals each led to two 
reports. The remaining 3 of the 91 FRP reports published by IES over this period did not have 
corresponding written initial proposals available from the Knowledge Utilization Division for panel 
review. 9 These three reports were themselves included in the panel review of reports, in order to 
represent fully the FRP reports published by IES by December 1, 2009. 

The study team grouped the 9 1 FRP reports and their corresponding proposals into 1 0 topic areas 
to assign them to content area expert panel members: (1) assessment, (2) standards, (3) staffing, (4) 
language arts and literacy, (5) math and science, (6) education policy, (7) at-risk populations, (8) 
school improvement, (9) data-driven decision-making, and (1 0) other. Methodologies represented in 
FRPs included descriptive and correlational studies, literature and document reviews, and qualitative 
studies. Many reports used mixed-methods approaches. 

The study team developed two rubrics to assist in the expert panel review of FRPs: one for IES- 
published reports and one for initial proposals (Appendix A). The draft rubrics underwent a series 
of external reviews and tests. First, the draft mbrics were reviewed by two TWG members for 
content accuracy and validity. In addition, they were pilot-tested by two groups in two content areas. 


8 It is important to note that the REL program contract start dates varied by REL as follows: REL Appalachia, February 6, 2006; REL 
Central, January 20, 2006; REL Mid-Adantic, March 23, 2006; REL Midwest, March 9, 2006; REL Northeast, March 15, 2006; REL 
Northwest, February 1, 2006; REL Pacific, March 16, 2006; REL Southeast, March 16, 2006; REL Southwest, March 15, 2006; and 
REL West, January 18, 2006. IES extended the contract end dates through December 31, 2011. 

9 These three reports were from REL Midwest, REL Southwest, and REL West. 
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Each group, comprising three members (two external and one internal to the study team), tested 
four products (two IES-published reports and two related initial proposals). Cognitive interviews of 
pilot-test reviewers were used to assess their understanding of the mbrics and rationale for the 
ratings. The external reviews and pilot-tests provided feedback used to further refine the rubrics. 

The final mbrics were designed to reflect and accommodate the heterogeneity of the FRPs. The 
rubrics included two dimensions: technical quality and relevance. Each dimension was further 
defined by multiple indicators. The mbrics for rating proposals and reports shared six indicators of 
quality and three indicators of relevance. Two additional indicators of quality and one additional 
indicator of relevance were included in the mbric for rating reports. The quality dimension 
indicators were as follows: 

1 . The research questions are clearly stated and are empirically testable. 

2. The data sources are appropriate for addressing the research questions. 

3. The population being studied and the sampling techniques (i.e., sample size, sampling 
strategies) are appropriate for addressing the research questions. 

4. Data collection instruments and their implementation are valid and reliable for the 
research questions. 

5. The analyses conducted are appropriate for the data used. 

6. All of the research questions are/will be adequately addressed. 

7. (reports only) The findings are clearly supported by the data. 

8. (reports only) The limitations of the project are clearly described. 

The relevance dimension indicators were as follows: 

1 . The report provides/ will provide information that can be used to inform decisions 
about policies, programs, or practices. 

2. The report contributes/will contribute new information to the topic being addressed. 

3. The report builds/will build on key literature and/ or previous research in the topic area. 

4. (reports only) The report is easy to read and understand given the intended audience. 

Indicators for the dimensions were rated on a 5-point scale, with 1 being the lowest rating and 5 
being the highest rating. Descriptive “anchors” were provided for the extreme and median points on 
the scale (values 1, 3, and 5). Indicators could also be rated as “not applicable” (NA) for a given 
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FRP. An indicator-specific comment field was provided to justify ratings of not applicable and to 
record rater’s notes, and a general comment field permitted reviewers to indicate uncertainty about 
the rating or note strength or weaknesses not linked to a specific indicator. 

The evaluation study team recruited 48 experts in content and/ or methodology, screened them for 
conflicts of interest, and trained them to use the rubrics for evaluating the quality and relevance of 
IES-published FRP reports and corresponding proposals. Panelists were primarily academics but 
also included contractors with expertise relevant to the content of the FRPs and/ or methods used. 
Forty of the 48 experts participated in a 1.5 day, in-person training in Houston, Texas, in January 
2010, and the remaining 8 experts participated in 1-day make-up training via WebEx in February 
2010. The specific purposes of the training were to inform expert panelists about their roles and 
responsibilities; teach them about the review process; help them become familiar with the scoring 
rubrics; and provide opportunities to practice scoring products using the mbrics, debrief, and 
undertake reconciliation with other experts. Comments from panelists during the training were used 
to make minor revisions to each mbric. 

A total of 79 panels were organized to review FRPs, including 46 experts and an average of 2.92 
experts per panel. The number of panels on which any one expert served averaged 5.02 and ranged 
from 2 to 10. Three panels reviewed no proposal, since no written initial proposal for the FRP was 
available. All other panels reviewed both the proposal for the FRP and the resulting IES-published 
report. Five panels reviewed multiple IES-published reports that arose from the same proposal. 
While three experts were generally assigned to review each set of FRP documents based on the 
experts’ content or methodology expertise, in five cases, fewer experts were assigned to review a 
product because no other panelists had the necessary expertise. In all but one instance, expert 
panelists submitted their ratings for each of the IES-published FRP reports and corresponding 
proposals assigned to them. Ratings for each FRP document (proposal or report) were generated by 
averaging the scores across all panelists and across dimension- specific indicators. 

To assess the reliability of the expert panel reviews, the study team measured inter-rater agreement 
by product type (i.e., proposals and IES-published reports) and by dimension (i.e., quality and 
relevance) using the “Rwg(j)” index (LeBreton and Senter 2008, see also Appendix B). Rwg(j) values 
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Table 2-1. Rwg (j) statistics for reviews of lES-published Fast Response Projects and 
corresponding proposals 




Dimension 


Product 

Quality 


Relevance 

Proposals (N= 75) 

0.92 


0.85 

Reports (N= 91) 

0.92 


0.85 


Table Reads: For the 75 proposals, the mean inter-rater agreement for indicator ratings along the quality dimension was 0.92. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 

Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 

Proposals) (Appendix A). 

range from 0 to 1 and provide estimates of the level of inter-rater agreement between sets of experts 
who provide ratings for different targets (i.e., IES-published reports and proposals in this case) using 
multiple items. Levels of Rwg® can be interpreted as follows (LeBreton and Senter 2008): 0.00-0.30 
(lack of agreement), 0.31-0.50 (weak agreement), 0.51-0.70 (moderate agreement), 0.71-0.90 (strong 
agreement), 0.91-1 (very strong agreement). 10 

While, in general, the initial inter-rater agreement was high (e.g., a mean Rwg® higher than 0.80), the 
study team held 1 1 reconciliation discussions with panels whose reviewed products had an Rwg less 
than 0.30, for a rate of 8 percent of all the products reviewed by the panelists. Five of the calls 
related to proposals and six to IES-published reports. Panel reviews and reconciliation phone calls 
occurred between February and May 2010. In reconciliation sessions, all three members convened 
over a conference call that was facilitated by a study team researcher. Panelists did not see the ratings 
of the other members of their panel, since confidentiality of ratings was assured to panelists. Only 
the dimensions and indicators with low agreement were reconciled. On the calls, experts were asked 
to discuss the strengths and weaknesses of the proposal or report relative to the indicator in 
question and were given an opportunity to revise their scores voluntarily after the call. Experts 
participating in reconciliation used the on-line rating system to modify their scores or informed the 
study team within 3 business days of the reconciliation call that no changes were being made. Three 
of the panelists chose not to change any of their scores after the reconciliation calls. All the others 
changed at least one of their scores. 

After reconciliation, mean inter-rater agreement for different types of reports and dimensions was 
high, with especially high mean levels of inter-rater agreement for the “Quality” dimension (Table 2- 
1). Further analysis of the Rwg(j) ratings indicated that they were negatively skewed and that the 


10 Additional information on the use of the Rwg(j) can be found in Appendix D. 
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median Rwg(j) is higher than the mean (e.g., “Quality” ratings on IES-published reports had a 
median Rwg(j) of 0.95 vs. A mean of 0.92.). 

Limitations of Expert Panel Review 

The expert panel review of FRP reports and proposals was characterized by several limitations in 
terms of the generaliz ability of the findings. FRP proposals that did not result in IES publications 
by December 1 , 2009, were not included in the expert panel review nor were draft FRP reports or 
reports published after December 1, 2009, included in the review. In addition, the expert panels 
organized for this report did not review other REL technical assistance products or dissemination 
events (such as Bridge Events). A list of the IES-published FRP reports included in the expert panel 
review is provided in the REL-specific chapters of the report. 
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Cross-REL Findings on the Quality and 
Relevance of Fast Response Projects 



This chapter summarizes the ways in which the RELs addressed their missions and focuses in 
particular on the technical quality and relevance of IES-published FRP reports and corresponding 
proposals. 

What activities did the RELs undertake to fulfill the missions speciGed in ESRAP 

In general, the RELs reported activities under each of the 10 statutory missions for the REL 
program, including those missions that were not explicit in the tasks of the statement of 
work for the REL contracts. Due to the semi-structured and open-ended nature of the data 
collection from REL staff to describe REL activities, the specific activities reported by each REL are 
documented only in the REL-specific chapters of this report, and results are not summarized across 
RELs. Chapters 4 through 13 of this report present, by REL and for each mission specified in 
ESRA, the specific activities described by REL directors in interviews with the evaluation study team 
between May 2010 and July 2010. 

What were the technical quality and relevance of the REL Fast Response Project reports 
published by IES and of the corresponding proposals? 

In completing an FRP, RELs typically submitted an initial proposal to the NCEERA Knowledge 
Utilization Division. The NCEERA Knowledge Utilization Division project officer could request 
changes or send the FRP proposal directly for external review through the ATS contractor. The 
review by ATS contract personnel could lead NCEERA to reject a proposal, accept it, or request 
revisions by the REL. Likewise, draft reports based on accepted proposals for FRPs were reviewed 
by project officers and other staff of the Knowledge Utilization Division and by the ATS contractor. 
Each level of review could result in NCEERA rejecting the report, accepting it for publication by 
IES, or requesting revisions by REL staff. Approval of the NCEERA commissioner was required 
before an FRP report was published by IES. There were no limits on the number of FRPs that a 
REL could propose to IES. However, each REL had an expected number of FRP products that was 
negotiated with the NCEERA project officer. To be approved under a REL contract, a project 
needed, in the judgment of the NCEERA, to address regional needs and be feasible to perform both 
technically and within the REL’s budget. Proposals for FRPs were typically included in RELs’ 
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contract proposals and as part of the RELs’ annual plans, although additional FRP proposals were 
introduced by the RELs at other times as well. 

Of 297 proposals for FRPs reviewed by NCEERA by December 1, 2009, 46 percent (137) were 
accepted, and the REL was authorized to proceed with the project. Of 166 reports from FRPs 
reviewed by IES by December 1 , 2009, 55 percent (92) were accepted for publication as IES reports. 
The number of proposals for FRPs submitted by each REL ranged from 17 to 45, and the 
acceptance rate for each REL ranged from 24 to 67 percent. The number of reports submitted for 
possible IES publication by each REL varied from 10 to 27, and the acceptance rate ranged from 25 
to 80 percent. Acceptance rates for each REL are included in the REL-specific chapters of this 
report. 

Independent panels of technical and content experts rated the technical quality and relevance of 91 
FRP reports published by IES by December 1, 2009, as well as the 75 initial proposals that led to 88 
of these reports. 11 Panelists rated quality and relevance using indicators on a 5-point scale, with 5 
being the highest value. 

The average quality ratings for the 91 FRP reports and 75 FRP proposals were 3.81 and 3.24, 
respectively. These ratings fell between “adequate” and “strong” on the 5-point quality 
scale. For proposals, 9.1 percent of quality ratings were “very strong” (5), while for IES-published 
FRP reports, 26.9 percent of quality ratings were “very strong” (Figure 3-1). Indicator-specific 
ratings of quality are provided in Figure 3-2. 

The average relevance ratings for the 91 FRP reports and 75 FRP proposals were 3.64 and 
3.39, respectively. These ratings fell between “adequate” in relevance and “relevant” on the 
5-point relevance scale. For proposals, 13.0 percent of relevance ratings were rated “highly 
relevant,” compared with 19.3 percent for IES-published reports (Figure 3-1). Ratings of IES- 
published FRP reports and corresponding proposals by specific indicators of technical quality or 
relevance are provided in Figure 3-2. 


11 Of the 92 reports counted as approved for publication by December 1, 2009, one report was not published until January 2010 and 
was not included in an expert panel review. Three of the 91 FRP reports published by IES by December 1, 2009, were without a 
written initial proposal available for panel review. 
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Figure 3-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from all 10 
RELs combined 
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Figure Reads: 9.1 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on 
a 5-point scale. 

NOTE: The distributions for proposal quality and relevance were based on 1,280 and 656 indicator-level ratings, respectively, which are 
largely attributed to differences in number of indicators associated with quality and relevance. The distributions for report quality and 
relevance were based on 2,051 and 1,065 indicator-level ratings, respectively. The difference in number of indicators associated with 
quality and relevance between reports and proposals is due to the fact the number of reports was larger than the number of proposals 
(see page xv). Ratings of “not applicable” were not included in these frequency distributions. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 

Readers are cautioned that it is not possible from the expert panel reviews of IES-published FRP 
reports and initial proposals to distinguish the contributions of the REL, the ATS contractor, or 
NCEERA to the quality or relevance ratings for the FRP reports published by IES. Readers should 
also be aware that, although there is correspondence between the 91 FRP reports reviewed and the 
75 initial proposals that resulted in 88 of those reports, the average indicator-level quality and 
relevance ratings are distributed differently for reports than for proposals, for two reasons. First, 
because some initial proposals led to more than one report, and three reports were without a 
corresponding written proposal, the average ratings are distributed differently across FRPs for reports 
and for proposals. Second, because the rubric for reviewing reports added two additional indicators 
of quality, and one additional indicator of relevance, to the six quality indicators and three relevance 
indicators in the mbric for rating proposals (Figure 3-2), the average ratings are distributed 
differently across indicators for reports and for proposals. 


The separate chapters for each REL discuss detailed REL-specific expert panel review findings. 
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Figure 3-2. Mean quality and relevance ratings of lES-published Fast Response Project reports 
and corresponding proposals, by rating indicator 
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Figure Reads: For indicator 1A, proposals received a mean rating of 3.48, and reports received a mean rating of 3.79. 

NOTE: Legend for Figure 3-2 (indicators marked with * are unique to reports). 

1A: The research questions are clearly stated and are empirically testable. 

IB: The data sources are appropriate for addressing the research questions. 

1C: The population being studied and the sampling techniques (i.e., sample size, sampling strategies) are appropriate for addressing the 
research questions. 

ID: Data collection instruments and their implementation are valid and reliable for the research questions. 

IE: The analyses conducted are appropriate for the data used. 

IF: All of the research questions can be/are adequately addressed. 

1G: The findings are clearly supported by the data.* 

1H: The limitations of the project are clearly described.* 

2A: The report provides information that can be used to inform decisions about policies, programs, or practices. 

2B: The report contributes new information to the topic being addressed. 

2C: The report builds on key literature and/or previous research in the topic area. 

2D: The report is easy to read and understand given the intended audience.* 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: Appalachia 



Regional Educational Laboratory (REL) Appalachia is one of 10 RELs nationwide authorized under 
the Education Sciences Reform Act (ESRA, P.L. 107-279) to support applied research, development, 
wide dissemination, and technical assistance activities that serve the educational needs of designated 
regions. Like the other RELs, REL Appalachia is funded through the Knowledge Utilization 
Division of the National Center for Education Evaluation and Regional Assistance (NCEERA) 
within the U.S. Department of Education’s Institute of Education Sciences (IES). 

REL Appalachia serves a region that includes the following states: 

• Virginia; 

• West Virginia; 

• Tennessee; and 

• Kentucky. 

For the 2006-201 1 contract period, REL Appalachia was housed at the CNA Institute for Public 
Research, a nonprofit research firm in Alexandria, Virginia. The FY 2006 award was the first time 
CNA had held a REL contract. 

Description of Projects and Activities 

In May 2010, a member of the evaluation study team interviewed the REL Appalachia director and 
two REL staff members. REL staff were asked to provide up to three examples of the work their 
REL had completed under each mission specified in ERSA. 12 Other than documenting REL reports 
published by IES, the evaluation study team made no attempt to verify independently the reported 
information on REL activities, so the accuracy and completeness of this information is unknown. 


12 For a more detailed description of the methodology, see chapter 2. 
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Mission #1: Training or Technical Assistance 

REL Appalachia identified four major technical assistance activities, although it noted that all its 
activities could be considered technical assistance. REL Appalachia staff highlighted the following 
projects and activities occurring under this mission: 

• Field Analysts. REL Appalachia assigned a research analyst to be responsible to assist each 
state department of education in the office of the commissioner (or close by). REL 
Appalachia reported that these field analysts had rigorous training in analysis and scientific 
method and worked with state agency personnel and educators across the state to help them 
think differently about education issues that have high priority in the state. For example, 
Tennessee was moving from a high-stakes exit exam to end-of-course exams. The state 
needed an index that categorized how each district rated under one system versus the other 
to answer the question: Is the change having disproportionate effects on certain types of 
schools? The REL Appalachia field analyst offered suggestions for different ways in which 
the index could be developed. 

• Bridge Events. At the time of the interview, REL Appalachia had conducted Bridge Events 
(as defined in Table 1-4) on the following five topics: 

1 . Dropout Prevention; 

2. Assisting Students Struggling with Reading: Response to Intervention (Rtl) and 
Multi-Tier Intervention in the Primary Grades; 

3. Dropouts in Tennessee and Strategies for Prevention; and 

4. Turning Around Chronically Low-Performing Schools. 

• Growth Models and Value-Added Assessments. REL Appalachia received a request 
from the Virginia Department of Education for a national expert on value-added 1 ' 
assessment. REL Appalachia prepared a Reference Desk response on value-added models 
and used its video teleconferencing facility to bring the team from the Virginia Department 
of Education together with representatives from three other states and nationally known 
experts. Following the event, REL Appalachia conducted follow-up activities with each of 
the states. Virginia and West Virginia subsequently communicated with one of the national 
experts about how to design their models. 


13 Specific examples of REL Appalachia’s work on value-added are located at http://www.cna.org/centers/education/selected- 
projects/ Vanderbilt. 
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• Special Projects. REL Appalachia characterized special projects as responses to specific 
requests for information from state or local officials. One example of a special project 
focused on universal access and public funding for pre-K. Constituents in West Virginia 
wanted to know what pre-K models were being used in the region. In response to the 
request, REL Appalachia did a quick-turnaround analysis and produced an annotated 
bibliography. At the time of the interview, the REL Appalachia staff members indicated that 
they had subsequently submitted a proposal to IES for a Fast Response Project (FRP) on 
pre-K to be conducted in 2011. 

Mission #2: Dissemination 

REL Appalachia staff members reported that they view dissemination as a form of two-way 
communicadon with their customers. As the REL director noted, it is not just a matter of sending 
materials (i.e., publications) to its customers. It is also trying to figure out how to get feedback about 
regional needs. REL Appalachia staff members added that successful dissemination requires meeting 
with customers face-to-face. 


REL Appalachia highlighted three tools it used to disseminate information: publications and 
products (examples of which are described below), the Internet, and in-person or face-to-face 
meetings with stakeholders. REL Appalachia reported that it tries not to send people a lot of long 
documents. Instead, the REL tries to target material to the appropriate groups. REL Appalachia 
staff members also noted the benefit of having local intermediaries, such as district superintendents, 
telling customers that the REL is a reliable source of information. REL Appalachia staff highlighted 
the following projects and activities occurring under this mission: 


• Publications and Products. In addition to the IES-accepted reports, REL Appalachia 
produces a newsletter, information bulletin, and occasional e-blasts (e.g., emails sent 
simultaneously to large groups). In the newsletter, the REL selects an education issue and 
provides one or two pages on each of the states’ activities on the specific issue, as well as 
references relevant National Center for Education Statistics (NCES) or What Works 
Clearinghouse publications, and other research links. The REL Appalachia information 
bulletin is distributed monthly and provides information about research newly released by 
IES. 

• Internet. REL Appalachia has an email distribution list that allows it to target recipients of 
materials by role (e.g., principals) and by state. In addition, REL Appalachia had just updated 
its website at the time of the interview so that it could be used for video teleconferencing. 
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• Face-to-Face Meetings. REL Appalachia staff members attend a variety of meetings, 
including all the state school superintendent association meetings for all four states in the 
region, to foster relationships with customers and support dissemination. 

Mission #3: Needs Assessment or Needs Sensing 

REL Appalachia used a composite of information from different sources to gauge the needs of its 
constituents. The REL staff members said that they informed their activities by reviewing national 
trends, state trends (particularly in the state legislatures), and local education agency (LEA) trends. 
The staff members also stated that they obtained a more “grounded” view by talking to customers in 
the region. 


REL Appalachia reported that at the start of the contract, it developed a list of priorities that drove 
the studies and refined the list over time. The REL proceeded to a monitoring phase to ensure that 
the work continued to be relevant to the region. For example, the REL staff members noted that 
since Tennessee won a Race to the Top (RTT) award, 14 the REL’s research took on added 
significance. In response, REL Appalachia worked with the Tennessee State Collaborative on 
Reforming Education (SCORE) (described under Mission #6), which is a nonprofit multi-sector 
education reform group that monitors and supports implementation of RTT in Tennessee. REL 
Appalachia staff highlighted the following projects and activities occurring under this mission: 


• Data Mining. Publicly available data — on demographic trends and student achievement, for 
example — provided REL Appalachia with background information to inform its work and 
suggested areas of regional need/interest. For example, at the time of the interview, the REL 
Appalachia staff members said that they had recently examined the characteristics of mral 
schools and districts. 

• Monitoring. The REL staff members reported that when REL Appalachia started its needs- 
sensing activities, it examined broad needs of the states in the region and gained much of its 
original information through formal discussions and focus groups with regional 
constituencies — regional officials, school superintendents, and principals. According to REL 
staff members, after gaining understanding of those broad needs, the REL investigated the 
details through a number of sources. REL Appalachia monitored what was reported and 
written on education-related issues in each of the four states. The REL’s governing board 11 
also served as a needs-sensing body, as did the field staff members embedded with state 
education agencies (SEAs) who routinely provided reports, not only on analysis, but also on 
the content of state board meetings and trends and issues in the four states. 


14 The U.S. Department of Education awarded Tennessee a Race to the Top grant on March 29, 2010 (see: 
http:/ / www2.ed.gov/ programs/ racetothetop/ awards.html). 

15 Each REL has a governing board that is designed to help identify and prioritize the educational needs of the region, review plans 
and products, and assist with outreach and dissemination efforts. 
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Mission #4: Carrying Out Applied Research Projects 

REL Appalachia staff highlighted the following projects and activities occurring under this mission: 

• Impact Studies. At the time of the interview, REL Appalachia was conducting two impact 
studies. The first study. The Effects of Opening the World of Teaming (OWE) on the Early Literacy 
Skills of At-Risk Urban Preschool Students , evaluated whether a research-based preschool 
curriculum increases the literacy achievement of children at risk. The study used a 
randomized pretest-posttest design in a large urban district in Tennessee, with students and 
teachers as units of observation. The second study, The Effects of Hybrid Algebra I on Teaching 
Practices, Classroom Quality, and Adolescent Teaming, tested whether a hybrid Algebra I program 
improves classroom quality and increases student engagement and achievement. 16 The 
hybrid program used online resources in face-to-face technology-enhanced classrooms to 
facilitate the use of standards-based instructional practices. Participating teachers engaged in 
sustained professional development focusing on effective pedagogy and the use of 
technology. The randomized study used a two-cohort research design. 

• Issues and Answers Reports. At the time of the interview, REL Appalachia had published 
seven FRP reports and had an additional six of these reports underway. 

• “Real Time” Analysis. REL Appalachia reported that it performed “real time” (quick- 
turnaround) analysis in partnership with a number of the universities in the region. 

Mission #5: Providing Applied Research Projects in a Usable Format 

REL Appalachia staff highlighted the following projects and activities occurring under this mission: 

• West Virginia Pre-K This project, mentioned previously under Mission #1 , provided 
information in a variety of formats, including in-person events and publications written for 
a lay audience. 

• English Language Learners (ELL). REL Appalachia’s work on ELL student registration 
began as part of a report and was later turned into a practice guide and webinar. The REL 
staff members reported that because ELL registration was identified as an area of need in 
the region, but few school officials knew how to track and monitor the educational progress 
of immigrant and ELL students, the REL printed a large number of guides. 

• Reference Desk. The REL staff members reported that they had completed many 
Reference Desk requests during the contract period. In work on the Reference Desk, REL 
Appalachia staff members indicated that they produced materials aimed at helping 
constituents understand the research and present information in a way that can be used in 
decision-making. REL Appalachia synthesized the research in brief documents, limiting the 
length to three pages. 


16 IES published a report from this study in April 2012. 
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Mission #6: Coordination and Collaboration 


In addition to the collaboration described below, REL Appalachia established formal partnerships 
with two types of organizations: (1) institutions of higher education (THEs) and other organizations 
with regional presence and depth in their respective states and communities; and (2) subject matter 
experts, such as Child Trends, the Rural School and Community Tmst, and the Center for Applied 
Linguistics. REL Appalachia staff highlighted the following projects and activities occurring under 
this mission: 


• SEAs and LEAs. According to the REL staff members, state education agencies were their 
most prominent partners as they worked to build regional capacity to conduct research and 
analysis. The lab also built on its relationships with SEA program managers to generate 
relationships and cooperative partnerships with local school district officials. Field scientists 
enabled REL Appalachia to maintain direct and ongoing relationships (one on one) with 
key SEA officials, including state superintendents. REL Appalachia also worked at the local 
education agency level, contacting LEAs through locally based national associations, such as 
the Council of Great City Schools, American Association of School Superintendents, and 
National Secondary School Principal Association. For example, the REL staff members 
reported working with a county school district in West Virginia with a strong interest in 
developing a research-based teacher assessment and professional learning community. The 
staff members noted that while REL Appalachia worked specifically with the LEA, it was 
also with the intent to carry the work over into other districts. 

• Community of Stakeholders. REL Appalachia worked with stakeholders and partners in 
the larger community, such as university researchers and education reformers. For example, 
the REL staff members reported that they established a collaborative relationship with the 
Tennessee State Collaborative on Reforming Education (SCORE) to work with state and 
local governments in Tennessee to encourage sound policy decisions in public education as 
well as to advance innovative reform on a statewide basis. The REL staff members noted 
that SCORE includes the chief state school officer, business leaders, university presidents, 
and other drivers of change. REL Appalachia worked to guide SCORE toward available 
relevant research to help inform its decisions. 
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• Cross -REL Collaboration. REL Appalachia cited several examples of its involvement in 

cross-REL collaborations. The REL reported it served as the leader of the REL data 
working group and played a large role in the mral working group. REL Appalachia co- 
hosted an American Recovery and Reinvestment Act (ARRA) event with several other labs 
(and the ED-funded Comprehensive Centers) and also co-produced a presentation at the 
NCES Management Information Systems conference. In addition, REL Appalachia 
described a proposal that involves all 10 RELs to study teacher effectiveness. The study 
addressed two questions: (1) How do states define teacher effectiveness, and (2) What data 
do they collect? REL Midwest served as the lead organization for this collaborative effort. 

Mission #7: School Finance Systems 

REL Appalachia staff highlighted the following projects and activities occurring under this 
mission: 

• Changes in the Cost of Energy in One State’s School Districts (FRP). This FRP 
documented energy expenditures in Tennessee school districts from 2002-03 to 2007-08 to 
document how much district spending on energy rose and whether the increase 
disproportionately affected districts with certain characteristics. At the time of the 
interview, the REL staff members reported that they were proposing an update to this 
study. 

• Instructional Funding. REL Appalachia proposed a study in 2009 that addressed ways 
states determine the percentage of funding that is allocated to instructional activities. The 
REL staff members noted that several state legislatures passed laws requiring that a 
specified percentage of funds be spent on classrooms, but SEAs need information on ways 
to determine these percentages. Although IES did not accept the initial FRP proposal, at 
the time of the interview, the staff members said that the proposal was being revised and 
resubmitted to IES. 

• Post-Recovery Act Education Finance. At the time of the interview, the REL staff 
members were in the early stages of work with states to plan for when the federal Recovery 
Act funds run out. 

Mission #8: Alternative Administrative Structures 

REL Appalachia staff highlighted the following projects and activities occurring under this 
mission: 
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• Hybrid Math (Impact Study). REL Appalachia conducted an impact study of hybrid 
math (described under Mission #4) that the REL staff members reported has implications 
for administrative structures. Specifically, if districts implement hybrid delivery (i.e., a mix 
of web/ traditional classroom instruction) on a larger scale, administrative structures may 
change to accommodate the new service delivery model. 

• Pre-K Collaboration. In the REL Appalachia region, pre-K is administered by a variety of 
agencies (e.g., education, health and human services). REL Appalachia conducted a study 
to document state laws and regulations related to pre-K and how they are implemented. 
Kentucky was on the verge of adopting legislation regarding pre-K, so REL Appalachia put 
this information together as technical assistance that initially served Kentucky but that was 
also used by other states. 

• Superintendent Turnover in Kentucky (FRP). At the time of the interview, the REL 
staff members were working on a study that examined superintendent turnover in 
Kentucky with an explicit focus on rural school districts and their communities. The report 
examines differences in turnover among communities with varying demographic and 
socioeconomic characteristics. It also explores variations in the characteristics of school 
districts experiencing different levels of turnover. 17 

Mission #9: School Improvement Plans 

REL Appalachia staff highlighted the following projects and activities occurring under this 
mission: 

• Turning Around Chronically Low-Performing Schools. REL Appalachia developed a 
page on its website with papers and links to resources on turning around chronically low- 
performing schools. The web page complemented the five-part webinar series on the same 
topic. The REL reported it has followed up by having specific conversations in each state 
around these issues. 

• Other. REL Appalachia staff members said that the REL believes its work has informed 
the development of good school improvement plans. According to the staff members, 

REL Appalachia did a Bridge Event in Nashville, Tennessee, on the response to 
intervention in early reading, an aspect of school improvement. REL Appalachia has also 
worked with districts to strengthen professional learning communities as part of a 
statewide school improvement plan. 

Mission #10: Technology in Education 

REL Appalachia staff highlighted the following projects and activities occurring under this 
mission: 


17 IES published a report from this study in August 2011. 
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• Hybrid Math (Impact Study). Hybrid math, described earlier under Mission #4, deals 
directly with issues of technology as it explores the impact of online delivery of 
mathematics instruction. 

• Technology-Based Programs for Instructing ELL Students. At the time of the 
interview, REL Appalachia was working on an FRP on ELL technology and resources to 
help customers understand the use of technology in the classroom to support ELL 
students. The FRP focuses on the following questions: (1) What are the uses of 
technology-based resources for ELLs in districts? (2) How do districts support or hinder 
the use of technology-based resources for instruction of ELLs? and (3) What types of data 
are available to district administrators in reporting on the use of technology-based 
resources? 

• REL Appalachia Technology Usage. REL Appalachia has undertaken a variety of 
activities to increase use of technology in its research-to-practice Bridge Events and in 
disseminating research-related information. The REL reported that these technology tools 
are particularly useful because many of the states in the region have no travel funds 
available. For example, the REL has established an online professional development 
platform. In this application, the software guides the user through the site materials, 
including video footage. It provides links to referenced documents throughout, so users 
can locate additional materials. 

Quality and Relevance of Fast Response Projects 

As of December 1, 2009, 37 percent (11) of REL Appalachia’s 30 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 4-1). Thirty-one percent (4) of REL Appalachia’s 13 submissions for FRP 
reports were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 


By December 1, 2009, REL Appalachia had four FRP reports accepted and published by IES: 

1 . Registering Students from Language backgrounds Other Than English (September 2007); 

2. Assessing the Likelihood That Virginia Schools Will Meet the Prof deny Goals of the No Child Left 
behind Act (September 2007); 

3. Preparing to Serve English Language Learner Students: School Districts With Emerging English 
Language Learner Communities (June 2008); and 

4. West Virginia’s Progress Toward Universal Prekindergarten (April 2009). 
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Table 4-2 shows the expert panel quality and relevance ratings for these IES-published reports from 
REL Appalachia and for the three corresponding proposals. 18 Average quality and relevance ratings 
for proposals and reports from all 10 RELs are provided for reference. Quality rating categories 
ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged from 1 
(“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 


Table 4-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Appalachia and all 10 RELs combined as of December 1, 2009 



Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 

Appalachia 

30 

11 

37 

All RELs 

297 

137 

46 

Reports 

Appalachia 

13 

4 

31 

All RELs 

166 

92 

55 


Table Reads : Of the 30 REL Appalachia proposals submitted, 11 (37 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and IES-published 
reports received a mean quality rating of 3.81 . FRP proposals from REL Appalachia received a mean 
quality rating of 3.38, and the corresponding IES-published reports received a mean quality rating of 
4.06. Among the three REL Appalachia proposals, mean quality ratings ranged from 3.17 to 3.72. 
Among the four REL Appalachia IES-published reports, mean quality ratings ranged from 3.50 to 
4.71. 


18 One proposal resulted in two reports, as noted in Table 4-1. Appendix A describes in detail the indicators that make up the quality 
and relevance dimensions. 
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Table 4-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Appalachia and all 10 RELs 
combined (on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All REL FRP proposals (N = 75) 

3.24 

3.39 

All Appalachia FRP proposals (N = 3) 

3.38 

3.52 

Assessing the Likelihood That Virginia Schools Will Meet 
the Proficiency Goals of the No Child Left Behind Act 

3.72 

2.89 

Preparing to Serve English Language Learner Students: 
School Districts With Emerging English Learner 
Communities; and Registering Students From Language 
Backgrounds Other Than English 

3.17 

3.33 

West Virginia’s Progress Toward Universal 
Prekindergarten 

3.25 

4.33 

Reports 


All REL FRP reports (N = 91) 

3.81 

3.64 

All Appalachia FRP reports (N = 4) 

4.06 

4.03 

Assessing the Likelihood That Virginia Schools Will Meet 
the Proficiency Goals of the No Child Left Behind Act 

4.71 

4.00 

Preparing to Serve English Language Learner Students: 
School Districts With Emerging English Language Learner 
Communities 

3.50 

3.75 

Registering Students From Language Backgrounds Other 
Than English 

4.00 

4.25 

West Virginia’s Progress Toward Universal 
Prekindergarten 

4.06 

4.12 


Table Reads: For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of FRP lES-published reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. 

SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 


With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published FRP reports by December 1, 2009, and a mean rating of 
3.64 for the actual IES-published reports. REL Appalachia proposals received a mean relevance 
rating of 3.52, and the corresponding IES-published reports received a mean relevance rating of 
4.03. Among the three REL Appalachia proposals, the mean relevance ratings ranged from 2.89 to 
4.33. Among the four REL Appalachia IES-published reports, the mean relevance ratings ranged 
from 3.75 to 4.25. Additional detail on the distribution of expert panelist indicator-level quality and 
relevance ratings is provided in Figure 4-1. 
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Figure 4-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Appalachia 
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Figure Reads: 8.3 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on a 
5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 48 and 24 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 81 and 44 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 3 

N (Reports) = 4 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: Central 



Regional Educational Laboratory Central serves a region that includes the following states: 

• Colorado; 

• Kansas; 

• Missouri; 

• Nebraska; 

• North Dakota; 

• South Dakota; and 

• Wyoming. 

For the 2006-201 1 contract period, REL Central was housed at Mid-continent Research for 
Education and Learning (McREL), a nonprofit organization located in Denver, Colorado. At the 
time of data collection, McREL had held the REL contract continuously since 1966. 

Description of Projects and Activities 

In May 2010, two members of the evaluation study team interviewed the REL Central director and 
two REL staff members. REL staff were asked to provide up to three examples of the work their 
REL had completed under each mission specified in ESRA. 19 Other than documenting REL reports 
published by IES, the evaluation study team made no attempt to verify independently the reported 
information on REL activities, so the accuracy and completeness of this information is unknown. 


19 For a more detailed description of the methodology, see chapter 2. 
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Mission #1: Training or Technical Assistance 

The REL Central staff members reported that they primarily served the administrators of state 
education agencies (SEAs) and their personnel, school districts, and school boards. These 
constituents contacted the REL and asked for guidance or assistance on education issues. REL 
Central responded to these requests via short-term technical assistance projects, such as responses 
from the REL Reference Desk. The REL team noted that it had completed approximately 38 short- 
term technical assistance projects. In some cases, the REL responded to the constituents’ questions 
with longer term projects. At the time of the interview, the REL team reported that it had completed 
12 longer term projects. In addition, REL Central had responded to constituents’ requests by 
conducting Bridge Events. REL Central staff highlighted the following projects and activities 
occurring under this mission: 

• Technical Assistance to the Nebraska Department of Education (NDE). NDE 

contacted REL Central and requested the REL’s assistance to analyze achievement gaps 
between student subgroups on the state assessment, including those for English language 
learners (ELLs) and highly mobile students. The REL helped the state to develop applicable 
measures and then delivered data to NDE broken out by school district for inclusion in the 
state’s report. 

• Technical Assistance to the Topeka School District. Topeka School District contacted 
REL Central and asked for the REL’s assistance to examine two different tests that the 
district was giving its ELL students, a state test and the IPT. The district was interested in 
learning if it was necessary to give the students both tests in order to predict performance. 
REL Central analyzed the two tests and reported to the school district that both tests were 
useful, but for different grade levels. 

• Technical Assistance to the Pueblo School District. Pueblo School District asked REL 
Central to evaluate district-wide data and work with principals to help design and monitor a 
school improvement process. 

Mission #2: Dissemination 

The REL director stated that the REL focused dissemination on products used throughout the 
region. For example, the REL focused on issues such as high school dropouts, American Indian 
education, and mral education. In addition to disseminating REL Central-specific products, such as 
a newsletter that discussed REL activities and reports, the REL also disseminated IES-published 
practice guides. The REL also used meetings as a means for dissemination. REL Central staff 
highlighted the following projects and activities occurring under this mission: 


20 http:/ /www.badard-tighe.com/products/la/iptFamilyTests.asp 


Evaluation of the Regional Educational Laboratories: Interim Report 

35 


• Website. REL Central’s website allowed constituents to download PDFs of the REL’s 
reports and received approximately 300 hits a month. 

• Bridge Events. REL Central hosted various Bridge Events based on the IES practice 
guides to disseminate information to its constituents. The REL held events on issues such as 
Response to Intervention (Rtl), literacy, and out-of-school time (OST). The OST event was 
requested by Nebraska’s education commissioner in conjunction with the Nebraska 
Community Learning Center Network." The event was based on an IES practice guide on 
OST, and one of the authors of the practice guide, a REL Central employee, spoke at the 
event. At the time of the interview, the REL had conducted Bridge Events on five topics: 

1. Turning Around Chronically Low-Performing Schools 

2. The National Math Panel Report: Recommendations, Lessons Learned, What’s 
Here for You? 

3. Bringing Research to Statewide Longitudinal Data Systems 

4. Structuring Out of School Time to Improve Student Achievement 

5. Assisting Students Struggling With Reading: Response to Intervention (Rtl) 

• Policy Forums. In collaboration with the Comprehensive Center also hosted by McREL, 
REL Central held an annual meeting called a Policy Fomm for the region. All SEA chiefs in 
the region were invited to attend. In addition, representatives from each state’s governor’s 
office and legislature were invited. REL Central reported that approximately three to five 
people attended from each state. Experts in certain education areas were invited to speak. 
Each meeting had a specific focus area, such as special education, restructuring, or state 
assessment/ accountability systems. During the meetings, SEA chiefs had the opportunity to 
discuss issues that affect their states, particularly emphasizing rural schools and finance. The 
meeting attendees also discussed questions such as. How might we create new policy? After 
the fomms were over, the REL distributed summaries of the events via the monthly 
newsletter. The REL Central director said that while these meetings were a form of 
dissemination, they also served as a mechanism for technical assistance and needs assessment. 

Mission #3: Needs Assessment or Needs Sensing 

In addition to formal needs assessments, REL Central conducted informal needs assessments when 
it held meetings, such as Bridge Events and policy fomms. At these meetings, the REL asked the 
participants what their needs were. REL Central also asked its Governing Board for feedback on 
regional needs. 


21 http: / / www.nebraskaclcnetwork.org/ 
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Once the REL collected needs assessment data from the region, it analyzed the data and created 
state profiles based on needs. According to the REL, these needs assessment results informed the 
REL’s technical assistance and training components. REL Central staff highlighted the following 
projects and activities occurring under this mission: 


• Gallup Organization Survey. REL Central received the Office of Management and 
Budget’s approval to have the Gallup Organization conduct a needs assessment of the region. 
Respondents were asked to rate 26 topics and indicate whether each issue was an important 
one for REL research or technical assistance. 

• Semi-Annual Meeting of the Chief State School Officers. The REL staff members met 
biannually with the chief state school officers in the region. During this meeting, the chiefs 
shared with the REL their states’ needs as well as how the REL could best meet those needs. 

• State Liaisons. REL Central assigned state liaisons to each state in the region. State liaisons 
were responsible for understanding the important issues in their state and connecting these 
issues to REL Central’s work. 

Mission #4: Carrying Out Applied Research Projects 

REL Central staff highlighted the following projects and activities occurring under this mission: 

• A Study of Classroom Assessment for Student Learning ( CASL) (Impact Study). The 

purpose of this randomized controlled trial was to examine the effect of CASL’s professional 
development on student achievement on a statewide achievement test. The primary research 
question was: Does teacher participation in CASL have a significant impact on student 
achievement? At the time of the interview, the REL staff members stated that this study was 
in its final stages. The REL had submitted a first draft of the report to ED’s Analytical and 
Technical Support contractor for review. The study was conducted in fourth- and fifth-grade 
classrooms for mathematics. REL Central reported that 67 public schools in Colorado 
participated in the study: 33 treatment schools and 34 control schools. 22 

• A Study of the Differential Effects of English Language Learner (ELL) Training and 
Materials (Impact Study). The purpose of this randomized controlled trial was to evaluate 
the effectiveness of the On Our Way to English (OWE) program for ELL students in 
conjunction with the Responsive Instruction for Success in English (RISE) program for ELL 
teachers. The study’s primary research question was: Does participation in RISE in 
conjunction with the use of OWE have a significant impact on the acquisition of English 
language skills? The secondary question was: Does the use of OWE in conjunction with 
RISE participation result in changed teacher pedagogical practices reflected in teacher 
behaviors and skills related to ELLs? At the time of interview, REL Central was finishing 


22 IES published a report from this study in April 2011. 
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data collection and said that a draft report would be submitted to IES at the end of July 2010. 
REL Central reported that a total of 48 schools in three states participated in the study. 23 

• A Study of the Effectiveness of a School Improvement Intervention (Success in Sight) 
(Impact Study). The study’s primary research question was: Does implementation of 
Success in Sight significantly improve student achievement? This study evaluated the 
effectiveness of a McREL product. Success in Sight, which is a comprehensive school 
improvement process. The REL staff members reported that in order to maintain objectivity 
and refrain from having a conflict of interest, an outside consulting firm did the evaluation 
and will write the report. McREL staff members, however, implemented the study. At the 
time of interview, this study was in the analysis stage. The REL staff members said that an 
interim report would be submitted to IES in early 2010. REL Central reported that 52 
schools participated in this study — 26 treatment schools and 26 control schools. The target 
population was low- to mid -performing elementary schools. 24 

Mission #5: Providing Applied Research Projects in a Usable Format 

REL Central provided applied research projects in usable formats to its constituents in several ways, 
such as Fast Response Projects (FRPs), shorter topical reports, and supplemental materials provided 
at Bridge Events or other meetings. The REL staff members reported that they created shorter 
documents (known as Research-in-Brief), that summarize the FRPs. The REL also created shorter 
documents to summarize the impact studies. REL Central staff highlighted the following projects 
and activities occurring under this mission: 

• A Description of Student Mobility in Rural and N onrural Districts in Five Central 
Region States (FRP). A group of district superintendents and school principals in rural 
areas in several constituent states requested that REL Central report on differences in 
mobility patterns between mral and non-mral locales. REL Central prepared maps that 
showed variations in mobility by school district and urbanicity. In addition, the REL 
compared average mobility rates by locale code for each state. At the time of the interview, 
the REL staff members noted that the report had been accepted by IES and would shortly be 
posted to the REL website. 

• High School Dropout and Completion Rates in the Central Region States (FRP). This 
study examined high school dropout and school completion rates in the each state in the 
Central region. 

• Examining American Indian Perspectives in the Central Region on Parent 
Involvement in Children’s Education (FRP). REL Central prepared a report that 
examined American Indian parents’ perceptions of parent involvement in their children’s 
education and factors that may encourage or discourage involvement. The REL staff 
members stated that to ensure that this report reached the constituents, supplemental 


23 IES published a report from this study in April 2012. 

24 IES published a report from this study in February 2012. 
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materials on the report’s findings were distributed at American Indian conferences. The REL 
also presented the report’s findings on several occasions. 

Mission #6: Coordination and Collaboration 


REL Central staff members reported that they coordinated and collaborated with other RELs, 
Comprehensive Centers, and substantive experts. REL Central participated with the other RELs in 
working groups (data, dissemination, rural, reference desk), Thursdays @ 3, REL director meetings, 
and the Reference Desk. The REL has also participated in cross-lab FRPs and has co-hosted 
webinars with other RELs on topics of mutual interest to the regions. REL Central staff highlighted 
the following projects and activities occurring under this mission: 


• American Recovery and Reinvestment Act (ARRA) Forums. In collaboration with 
several RELs, Comprehensive Centers, and Regional Resource Centers, REL Central co- 
sponsored two ARRA fomms that addressed ARRA funding streams and opportunities, 
Statewide Longitudinal Data System Funds, and Title I School Improvement Grants. REL 
Central reported that representatives from all seven states in its region attended these forums. 

• Cross-REL Projects. In collaboration with several RELs, REL Central participated in the 
FRP on the achievement gap for Native American students. Specifically, REL Central 
collected and analyzed data. At the time of the interview, the REL staff members said that 
they planned to develop six study briefs from the FRP that focus on the REL Central states. 

• Technical Working Group (TWG). REL Central staff members stated that through their 
own TWG, the REL collaborated with research experts in various areas (i.e., research design, 
statistics, psychometrics, etc.). Their TWG met annually to receive a general overview of all 
ongoing studies and to address methodological and content concerns for individual studies. 
The TWG members provided guidance on all of REL Central’s impact studies. 

Mission #7: School Finance Systems 


REL Central staff highlighted the following projects and activities occurring under this mission: 


Access to Supplemental Educational Services (SES) in the Central Region States 

25 

(FRP). SEA SES staff members requested that REL Central examine Title I program 
allocations to SES in the Central region because student participation in SES was below 
national averages. According to the REL staff members, the resulting report showed that 
mral schools and districts were having difficulty setting aside 20 percent of the Title I dollars 
for a program that did not appear to be working in isolated mral schools. 


• High-Poverty Districts. At the time of the interview, the REL Central staff members stated 
that they had recendy received a request from the state of Nebraska to develop a project to 
assist the state in determining the research basis of components of their poverty plans. 


25 http:/ /www2.ed.gov/nclb/choice/help/ses/description.html 
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According to the REL staff members, in Nebraska, schools must submit poverty plans by a 
specified deadline in order to receive funding. Officials from Nebraska requested that REL 
Central conduct a literature review to determine which factors are effective for the high- 
poverty schools. 

• Four-Day School Week. At the time of the interview, the REL Central staff members stated 
that they were working on a district request for assistance with examining the outcomes of 
going to a 4-day school week. According to the REL, this issue was of particular importance 
to small, rural districts moving to 4-day school weeks for cost savings. The REL reported 
plans to help the district design an evaluation and conduct the needed intermpted time series 
analysis to examine student attendance and possibly student achievement effects. 

Mission #8: Alternative Administrative Structures 

REL Central staff highlighted the following projects and activities occurring under this mission: 

• A Study of the Effectiveness of a School Improvement Intervention (Success in Sight) 
(Impact Study). Described in Mission #4, this impact study evaluated the effectiveness of a 
McREL product. Success in Sight, which is a comprehensive school improvement process. 

• Four-Day School Week. Described in Mission #7, REL Central staff members reported at 
the time of interview that they planned to assist a district with an interrupted time series 
analysis to study the effect of the 4-day school week on student attendance and possibly 
student achievement. 

• Identifying Differences Between Two Groups of High-Needs High Schools (FRP). At 

the time of the interview, the REL staff members were working on a study to examine 
differences in student achievement between a group of high-performing high-needs (HPHN) 
high schools and a group of low-performing high-needs (LPHN) high schools. The study was 
intended to determine whether specific factors (such as administrative factors) were related to 
student achievement in HPHN and LPHN high schools. 

Mission #9: School Improvement Plans 

REL Central staff highlighted the following projects and activities occurring under this mission: 

• High School Dropout and Completion Rates in the Central Region States (FRP). 

Noted in Mission #5, this study examined high school dropout and school completion rates 
in each state in the Central region. 

• Examining American Indian Perspectives in the Central Region on Parent 
Involvement in Children’s Education (FRP). Described in Mission #5, REL Central 
examined American Indian parents’ perceptions of parent involvement in their children’s 
education and factors that may encourage or discourage involvement. 

• A Study of the Effectiveness of a School Improvement Intervention (Success in Sight) 
(Impact Study). Noted in Mission #4 and Mission #8, this study evaluated the effectiveness 
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of a McREL product. Success in Sight, which is a comprehensive school improvement 
process. 

Mission #10: Technology in Education 

REL Central initially proposed two studies in the area of technology and education. Elowever, 
neither study moved forward. REL Central staff highlighted the following projects and activities 
occurring under this mission: 

• Describing the Use of Technology in Eighth-Grade Mathematics. This proposed study 
was designed to answer questions regarding student-reported use of computers in 
mathematics instruction. 

• Enhancing Education Through Technology (EETT). This proposed study was designed 
to examine how a specific school district was allocating its funding for the EETT program. 

Quality and Relevance of Fast Response Projects 


As of December 1, 2009, 40 percent (8) of REL Central’s 20 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 5-1). Forty-seven percent (7) of REL Central’s 15 submissions for FRP 
reports were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 


By December 1, 2009, REL Central had seven FRP reports accepted and published by IES: 

1 . Access to Supplemental Educational Services in the Central Region States (July 2007); 

2. Using Strategy Instruction to Help Struggling High Schoolers Understand What They Read (October 
2007); 

3. What States Can Team About State Standards and Assessment Systems From No Child Eeft behind 
Documents and Interviews With Central Region Assessment Directors (March 2008); 

4. High School Standards and Expectations for College and the Workplace (June 2008); 

5. High School Dropout and Graduation Rates in the Central Region (July 2008); 

6. Preparing Teachers to Teach in Rural Schools (July 2008); and 

7. Examining American Indian Perspectives in the Central Region on Parent Involvement in Children ’s 
Education (August 2008). 
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Table 5-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Central and all 10 RELs combined as of December 1, 2009 


Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 


Central 

20 

8 

40 

All RELs 

297 

137 

46 

Reports 


Central 

15 

7 

47 

All RELs 

166 

92 

55 


Table Reads : Of the 20 REL Central proposals submitted, 8 (40 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


Table 5-2 shows the expert panel quality and relevance ratings for these IES-published reports from 
REL Central and for the seven corresponding proposals. -6 Average quality and relevance ratings for 
proposals and reports from all 10 RELs are provided for reference. Quality rating categories ranged 
from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged from 1 (“not 
relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and IES-published 
reports received a mean quality rating of 3.81. FRP proposals from REL Central received a mean 
quality rating of 3.15, and the corresponding IES-published reports received a mean quality rating of 
3.64. Among the seven individual REL Central proposals, mean quality ratings ranged from 2.59 to 
3.47. Among the seven individual REL Central IES-published reports, mean quality ratings ranged 
from 2.92 to 4.80. 

With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published FRP reports by December 1, 2009, and a mean rating of 
3.64 for the actual reports. REL Central proposals received a mean relevance rating of 3.46, and the 
corresponding IES-published reports received a mean relevance rating of 3.43. Among the seven 
REL Central proposals, the mean relevance ratings ranged from 2.67 to 3.78. Among the seven REL 
Central IES-published reports, the mean relevance ratings ranged from 2.33 to 4.00. Additional 
detail on the distribution of expert panelist indicator-level quality and relevance ratings is provided in 
Figure 5-1. 


26 Appendix A describes in detail the indicators that make up the quality and relevance dimensions. 
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Table 5-2. Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Central and all 10 RELs combined 
(on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All REL FRPs (N = 75) 

3.24 

3.39 

All Central FRPs (N = 7) 

3.15 

3.46 

Access to Supplemental Educational Services in the 
Central Region States 

3.22 

3.67 

Examining American Indian Perspectives in the Central 
Region on Parent Involvement in Children’s Education 

2.78 

3.56 

High School Dropout and Graduation Rates in the Central 
Region 

3.47 

3.22 

High School Standards and Expectations for College and 
the Workplace 

3.28 

3.67 

Preparing Teachers to Teach in Rural Schools 

2.59 

3.67 

Using Strategy Instruction to Help Struggling High 
Schoolers Understand What They Read 

3.29 

3.78 

What States Can Learn About State Standards and 
Assessment Systems from No Child Left Behind 
Documents and Interviews With Central Region 
Assessment Directors 

3.44 

2.67 

Reports 

All REL FRPs (N = 91) 

3.81 

3.64 

All Central FRPs (N = 7) 

3.64 

3.43 

Access to Supplemental Educational Services in the 
Central Region States 

3.29 

3.75 

Examining American Indian Perspectives in the Central 
Region on Parent Involvement in Children’s Education 

3.21 

3.00 

High School Dropout and Graduation Rates in the Central 
Region 

4.80 

4.00 

High School Standards and Expectations for College and 
the Workplace 

3.43 

3.75 

Preparing Teachers to Teach in Rural Schools 

3.71 

3.83 

Using Strategy Instruction to Help Struggling High 
Schoolers Understand What They Read 

2.92 

2.33 

What States Can Learn About State Standards and 
Assessment Systems from No Child Left Behind 
Documents and Interviews With Central Region 
Assessment Directors 

4.12 

3.33 


Table Reads: For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 


Evaluation of the Regional Educational Laboratories: Interim Report 

43 


Figure 5-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Central 
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Figure Reads: 4.1 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on a 
5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 123 and 63 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 163 and 84 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 7 

N (Reports) = 7 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: Mid-Atlantic 



Regional Educational Laboratory Mid-Atlantic serves a region that includes the following 
jurisdictions: 

• Delaware; 

• District of Columbia; 

• Maryland; 

• New Jersey; and 

• Pennsylvania. 

For the 2006-201 1 contract period, REL Mid-Atlantic was based at the Pennsylvania State 
University (PSU) in University Park, Pennsylvania, and included four primary subcontractors: 
Rutgers University, ICF International, ANALYTICA, and the Metiri Group. Three of the REL Mid- 
Atlantic partner organizations had primary responsibility for TA and outreach for specific 
jurisdictions within the region. PSU worked with Pennsylvania; Rutgers University was responsible 
for New Jersey and Delaware; and ICF International worked with the District of Columbia and 
Maryland. The 2006-201 1 contract period was the first time PSU or any of its partner organizations 
held a REL contract. 

Description of Projects and Activities 

In May 2010, a member of the evaluation study team interviewed the REL Mid-Atlantic director 
and, at the director’s request, conducted a subsequent telephone interview with another of the REL 
staff members to fill in missing information. This follow-up interview was only to obtain more 
detailed information on REL Mid- Atlantic’s collaboration and coordination projects (i.e., Mission 
#6), which are primarily led by ICF International. In the interviews, REL staff members were asked 
to provide up to three examples of the work their REL had completed under each mission specified 
in ESRA. 2 Other than documenting REL reports published by IES, the evaluation study team made 


27 For a more detailed description of the methodology, see chapter 2. 
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no attempt to verify independently the reported information on REL activities, so the accuracy and 
completeness of this information is unknown. 

Mission #1: Training or Technical Assistance 

REL Mid-Atlantic staff highlighted the following projects and activities occurring under this 
mission: 

• Creating an Early Warning System: Predictors of Dropouts in Delaware (FRP). At the 

request of the Delaware Department of Education and Delaware’s P-20 Council, REL Mid- 
Atlantic prepared a Fast Response Project (FRP) on the key indicators of dropouts for 
Delaware students in grades 9-12. 

• A Descriptive Review of the Maryland State Department of Education ’s (MSDE) 
Science, Technology, Engineering, and Mathematics (STEM) Education Initiative 
Grants (FRP). The Maryland Department of Education asked REL Mid-Atlantic to review 
its STEM grants. Specifically, the department wanted the REL to (1) synthesize information 
across school systems and across three grant cycles to identify program characteristics and 
practices developed by Maryland school systems in projects funded through the grant and 
(2) provide information to the states as they prepare subsequent Requests for Proposals and 
refine the grantee reporting requirements. 

• The Predictive Validity of Selected Benchmark Assessments in the Mid-Atlantic 
Region (FRP). REL Mid-Atlantic examined the availability and quality of predictive validity 
data for a selection of benchmark assessments used within the Mid-Atlantic region, as 
identified by state and district personnel. 

Mission #2: Dissemination 

REL Mid-Atlantic reported that it disseminated information to its constituents in multiple ways, 
such as Bridge Events, laboratory extension specialists, the REL Mid-Atlantic website, internal list 
serve, and Ask-A-REL. REL Mid-Atlantic staff highlighted the following projects and activities 
occurring under this mission: 


• Bridge Events. At the time of the interview, REL Mid-Atlantic had held Bridge Events on 
26 topics. A list of the event titles are at the end of the Mission #2 section below. The REL 
director noted that REL Mid-Atlantic had 13 additional Bridge Events planned. At the 
REL’s Bridge Events, participants discussed the Bridge Event topic, in addition to hearing a 
presentation by the Bridge Event expert. REL Mid-Atlantic followed up with participants 
after Bridge Events via webinars, upon request. During the webinars, participants were able 
to converse with the presenter. The REL director noted that the REL valued maintaining 
communication with its constituents in this way, rather than hosting “one dose” Bridge 
Events. According to the REL director, in recognition of its rural constituents and school 
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districts’ budget constraints, REL Mid-Atlantic used video technology to hold the Bridge 
Events in multiple locations simultaneously. The titles of the Bridge Events were: 

1 . Improving Adolescent Literacy in NJ: Effective Classroom and Intervention 

2. Improving Adolescent Literacy in DE: Effective Classroom and Intervention 

3. Turning Around Chronically Low Performing Schools in DC 

4. Turning Around Chronically Low Performing Schools in MD 

5. Dropout Prevention in MD 

6. Improving Adolescent Literacy: Effective Classroom and Intervention Practice 

7. Organizing Instruction and Study to Improve Student Learning in PA 

8. Reducing Behavior Problems in the Elementary School Classroom in DE 

9. Reducing Behavior Problems in the Elementary School Classroom in NJ 

10. Turning Around Chronically Low Performing Schools 

11. Organizing Instruction and Study to Improve Student Learning: A Research Fomm 
for Mid-Atlantic Region Leaders 

12. Reducing Behavior Problems in the Elementary Classroom: Bridging the Gap 
Between Research and Practice 

13. Improving Adolescent Literacy: Effective Classroom and Intervention Practices, A 
Research Forum for Mid-Atlantic Region Leaders 

14. Informing Leadership at the School Level With Educational Research 

15. Structuring Out-of-School Time to Improve Academic Achievement in 
Washington, DC 

16. Encouraging Girls in Math and Science 

17. Assisting Students Struggling With Reading: Response to Intervention and Multi- 
Tier Intervention in the Primary Grades 

18. Research-Based Practices in K-12 Literacy 

19. Organizing Instruction and Study to Improve Student Learning 

20. Assisting Students Stmggling With Mathematics: Response to Intervention (Rtf) for 
Elementary and Middle Schools 

21. Dropout Prevention 

22. Using Student Achievement Data to Support Instructional Decision Making 

23. Effective Literacy and English Language Instruction for English Learners in the 
Elementary Grades 

24. Thinking Strategically Across American Recovery and Reinvestment Act (ARRA) 
Reform Initiatives 

25. Helping Students Navigate the Path to College: What High Schools Can Do 
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26. Structuring Out-of-School Time to Improve Academic Achievement 


• Laboratory Extension Specialists and Related Staff. The REL employed approximately 
17 part-time staff members, called laboratory extension specialists (LES), to connect with its 
constituents at the local education agency (LEA) and school levels and in professional 
associations. These staff members were often retired superintendents and principals. LES 
contacted each LEA approximately twice every year and visited with professional 
organizations such as school boards. According to the REL director, this task was 
challenging due to the large number of LEAs in the Mid-Atlantic. When speaking with 
constituents, LES explained the role of the REL and provided information on ways to 
contact the REL (e.g., Ask-A-REL). REL Mid-Adantic also employed state coordinators 
whose responsibility was to interact with personnel in the SEAs. In addidon, the REL 
director reported that the REL employed methodological experts, known as research 
extension specialists (RES), who were regionally dispersed. 

• Additional Dissemination Activities. The director described several other ways that REL 
Mid-Atlantic disseminated information. For example, it published documents and hosted a 
website designed to allow constituents to access recorded Bridge Events (video and audio). 
REL Mid-Adantic also had a list serve and a toll-free number that it used to disseminate 
information. REL Mid-Atlantic contributed to the Ask-A-REL database used to respond to 
constituent inquiries. 

Mission #3: Needs Assessment or Needs Sensing 

REL Mid-Atlantic conducted needs assessments in several ways. For example, the REL director 
noted that REL staff members met with SEAs, LEAs, and school boards to discuss their needs and 
how the REL could help them. REL Mid-Atlantic staff highlighted the following projects and 
activities occurring under this mission: 


• Laboratory Extension Specialists (LES). As described above, the REL director reported 
that LES were continuously in the field asking how REL Mid-Atlantic could best serve 
constituents. Identified needs were entered into a database and summarized to direct the 
majority of REL Mid-Atlantic’s technical assistance work. 

• Conversations With the Governing Board and SEAs. Twice a year, REL Mid-Atlantic 
assembled its Governing Board to discuss research needs. The REL director asked the Board 
to look to the future, specifically considering anticipated policy decisions and ways the REL 
could help present issues and/ or prepare members for these decisions. Similarly, the REL 
director stated that REL Mid-Atlantic visited the SEAs to ask them how the REL could best 
help them. 

• Ask-A-REL and Toll-Free Number (1-866-RELMAFYI). The REL director said that 
REL Mid-Atlantic had over 146 information requests from its constituents via Ask-A-REL. 
The director stated that the constituents who submitted questions represented a cross-section 
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of people, such as teachers, principals, curriculum coordinators, superintendents, SEA 
personnel, school board members, and, to a lesser degree, policymakers. 

Mission #4: Carrying Out Applied Research Projects 

REL Mid-Atlantic’s applied research projects comprised impact studies and FRPs. The first two 
projects highlighted below are impact studies and originated from requests from the field about how 
to identify effective math curricula. REL Mid-Adantic staff highlighted the following projects and 
activities occurring under this mission: 

• A Multisite Cluster Randomized Trial of the Effects of Compass Learning Odyssey 
Math on the Math Achievement of Selected Grade 4 Students in the Mid-Atlantic 
Region (Impact Study). According to the REL director, this study involved 70 schools and 
was the first study to assess the impact of Odyssey Math on student achievement that met the 
quality standards of the What Works Clearinghouse. The REL director said that REL Mid- 
Atlantic and its stakeholders chose to conduct an impact study on math curriculum as a result 
of a national push to improve math achievement. The study’s results showed that Odyssey 
Math did not have a statistically significant effect on end-of-year student achievement in 
grade four. 

• The Effects of Connected Mathematics 2 on Math Achievement in Grade 6 in the 
Mid-Atlantic Region (Impact Study). At the time of the interview, this 2-year study was in 
its second year. The sample consisted of 69 schools, 300 classrooms, and 7,000 students. The 
REL director reported that the study examined the effect of Connected Mathematics 2, a 
curriculum aligned with the National Council of Teachers of Mathematics standards that 
emphasizes authentic, relevant problem-solving practices to help students build math skills in 
a useful multidisciplinary context. Connected Mathematics 2 was used in intervention schools 
as the primary curriculum for grade six math during the 2008-09 (Year 1) and 2009-10 (Year 
2) school years. Control schools used their normally scheduled curricula for both study years. 
The primary outcome, student math achievement, was measured with pre- and post-tests 
using a standardized math test. 28 

• Identifying Selected Language Arts Literacy, Mathematics, and Science Instructional 
Materials in the Mid-Atlantic Region (FRP). According to the REL director, REL Mid- 
Atlantic SEAs had very little information regarding the instructional materials their school 
districts adopt. This report described first-year results of a project to share information on 
texts, materials, and assessments used by elementary, middle, and high school grade levels in 
language arts /literacy, mathematics, and science in the Mid-Atlantic region. Not only did this 
project result in an FRP report, but REL Mid-Atlantic also developed a searchable database 
of instructional materials used by districts across the region. The database could be accessed 
from the REL website. 29 


28 IES published a report from this study in March 2012. 

29 In order to conduct this FRP, the REL received Office of Management and Budget (OMB) approval. 
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Mission #5: Providing Applied Research Projects in a Usable Format 

REL Mid-Atlantic staff highlighted the following projects and activities occurring under this 
mission: 

• Fact Sheets. REL Mid-Atlantic produced “fact sheets” written for a lay audience. The fact 
sheets explained the REL research agenda, products, and services, including needs-sensing 
activities, impact studies, FRPs, and the services the LES provide. LES distributed the fact 
sheets during face-to-face visits to school districts, at conferences, and at other professional 
meetings. 

• Podcasts and “Trailer” Videos. In the update to its Year 5 plan, REL Mid-Atlantic 
proposed to develop brief videos and a series of audio podcasts to promote interest in REL 
work. At the time of the interview, REL Mid-Atlantic planned to post these videos and 
audios on the web page where the reports were posted. According to the director, the videos 
would be short, like a Hollywood “trailer,” while the podcasts would be longer. People would 
be able to subscribe to the podcasts in order to receive new podcasts automatically. The REL 
director reported that the overarching intent behind the podcasts and videos was to help 
readers better understand report content. REL Mid-Atlantic expected to conduct this work in 
cooperation with one or more other RELs. 

Mission #6: Coordination and Collaboration 

REL Mid-Atlantic staff highlighted the following projects and activities occurring under this 
mission: 


• Rural Working Group. A REL Mid-Atlantic employee created this working group in 2007 
and continued to lead the group through the time of the interview. The primary purpose of 
the working group was to share information about developments in rural education, build a 
stronger network among RELs around these issues, and identify opportunities for the RELs 
to serve rural constituents. This working group presented at the National Rural Education 
Association (NREA) conferences. In addition, the working group invited speakers from other 
organizations with a rural research and education focus. 

• Bridge Event on Response to Intervention With REL Southeast: “Using Research to 
Strengthen Response to Intervention (Rtl ) Decision Making and Implementation. ” 

REL Mid-Atlantic worked with REL Southeast to build attendance for this Bridge Event. At 
the event, REL Mid-Atlantic introduced panel members, led table group discussions, and 
provided videotaping services. After the event ended, REL Mid-Atlantic edited the videos 
and provided them to REL Southeast, so REL Southeast could post them on its website. 

• FRP Collaboration With REL Southwest. SEAs requested that REL Mid-Atlantic write a 
report to help them understand effective practices and policies in use for statewide data 
systems. Once REL Mid-Atlantic identified the four best state systems, it contacted REL 
Southwest, since two of the identified states were in that region, and developed a partnership 
with REL Southwest to collect data and write the report. 
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REL Mid-Atlantic was different from the other RELs in that it had the additional responsibility for 
cross-REL collaboration and coordination. Of the organizations that made up REL Mid-Adantic, 
ICF International, led the activities that fell under the collaboration and coordination task. Some of 
the collaboration and coordination activities included: 

• National Laboratory Network internet and intranet, 

• Six cross-REL working groups, 

• REL Reference Desk, 

• Ask-A-REL, and 

• Thursdays @ 3. 

The National Laboratory Network (NLN). The NLN Internet was housed on the IES website 
and was for public use. It contained web pages for each of the RELs and links to REL report 
publications. The NLN coordinated content updates and provided monthly website usage statistics. 
Alternatively, the NLN Intranet was also for internal REL use. It was primarily used for cross-REL 
collaboration and working group space. 

Working Groups. The six cross-REL working groups focused on different issues, such as data, 
dissemination, rural education, ARRA, and REL Reference Desk. The various working groups held 
periodic conference calls, and NCEERA Knowledge Utilization Division project officers sometimes 
joined as well. Additionally, there was dedicated space on the NLN Intranet for the working groups 
to share materials with members. Finally, a distribution list existed for each working group to allow 
for information dissemination. The level of REL participation and the specific types of activities 
varied by working group. 

REL Reference Desk. The REL Reference Desk housed responses to technical assistance 
questions and also had experts available to answer questions that had not been previously addressed 
by the RELs. The Reference Desk collaborated with the National Library of Education (NLE); 

RELs were able to contact the library for assistance in answering constituents’ questions. The NLE 
also provided the RELs with access to online journals. All RELs were able to use this resource to 
respond to their individual technical assistance requests. 

Ask-A-REL. This was a link on the REL website (hosted at IES) that allowed users to ask questions 
of specific RELs and receive corresponding responses (generally by email) from REL staff members. 
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Thursdays @ 3. These were bimonthly conference calls in which RELs discussed and collaborated 
on various issues. Anywhere from two to four people from each REL typically participated in 
Thursdays @ 3. 


In addition to its cross-REL coordination and collaboration activities, REL Mid-Atlantic also 
collaborated with non-REL entities, such as the Pennsylvania Association for Supervision and 
Curriculum Development, the New Jersey Council of Teachers of Mathematics, SEAs, and the Mid- 
Atlantic Comprehensive Center (MACC). The REL had a liaison that worked directly with MACC; 
the two organizations co-sponsored Bridge Events. 

Mission #7: School Finance Systems 

REL Mid-Atlantic staff highlighted the following projects and activities occurring under this 
mission: 

• Good Cents: State Strategies for Increasing Education Cost Efficiency in the Mid- 
Atlantic Region (FRP). The Associate Secretary for Administration and Innovation at the 
Delaware Department of Education asked REL Mid-Atlantic to supply the SEA with 
information about the cost-effectiveness measures other SEAs in the region were using. This 
study was underway at the time of the interview. Researchers were compiling these data 
through searches of publicly available websites and documents as well as structured 
telephone interviews with eight SEA personnel, two from each SEA that agreed to 
participate in the study. 

• Recruited Participants for REL Southwest Bridge Event, “Educator Staffing, 

Quality, and Teacher Retirement Benetit Systems. "To help REL Southwest recruit 
attendees for this Bridge Event, REL Mid-Atlantic recmited participants from the Mid- 
Atlantic region. The REL director noted that this Bridge Event topic was particularly 
relevant at the moment of the interview, since many school districts were struggling with 
how to finance teacher retirements. 

Mission #8: Alternative Administrative Structures 

REL Mid-Atlantic staff highlighted the following project or activity occurring under this mission: 

• The Fiscal and Academic Impacts of School District Consolidation (FRP). REL Mid- 
Atlantic met with the National Rural Education Research Center and the Pennsylvania Center 
on Rural Education to discuss research on the fiscal and academic impacts of school district 
consolidation. The director reported that the Pennsylvania governor has proposed creating a 
commission to study how best to “right-size” Pennsylvania’s local school districts. According 
to the director, the governor would like to have no more than 1 00 school districts in a state 
that currently has more than 500 school districts. This is a request that at the time of the 
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interview had not yet received IES acceptance. REL Mid-Atlantic proposed to synthesize the 
research on school district consolidations and potentially use student data systems to update 
previous research with new information. 

Mission #9: School Improvement Plans 30 

REL Mid-Atlantic staff highlighted the following projects and activities occurring under this 

mission: 

• Bridge Events on Turning Around Low-Performing Schools. At the time of the 
interview, REL Mid-Atlantic had presented five Bridge Events based on a What Works 
Clearinghouse Practice Guide, Turning Around Low-Performing Schools. These events were 
held in the District of Columbia, Maryland, New Jersey, and Pennsylvania. Experts presented 
at the Bridge Events, followed by panel discussions that featured practitioners who had 
experience turning around low-performing schools. Participants then took part in table 
discussions about how to make similar progress in their schools. 

• “Schools Like Mine ” database. According to the REL director, although IES did not 
release this product after several rounds of review, REL Mid-Atlantic developed an online 
tool that allowed school leaders and others to make cross-school comparisons of schools’ 
academic performance to determine whether their school was “doing well, for a school like 
ours.” 

Mission #10: Technology in Education 

REL Mid-Atlantic staff highlighted the following projects and activities occurring under this mission: 

• Key Issues in Creating a Statewide Virtual School: Paths Taken and Lessons Learned 
(FRP). The REL director reported that Pennsylvania recently passed legislation creating a 
Virtual High School Study Commission to investigate the creation of a “virtual high school” 
to provide students with educational services over the Internet. The commission requested 
that REL Mid-Atlantic produce a report that analyzes issues associated with establishing a 
statewide virtual high school to inform policy discussion and guide the development of new 
state virtual high schools in Pennsylvania and elsewhere. Drawing on data from four states 
that have statewide virtual schools, this report addressed three key questions: 

o Which options did education leaders consider; what decision did they make; and 
what were the reasons for choosing that position? 

o How well are those decisions serving the school today? 

o What changes are being considered, and what forces are creating the need to 
consider making these changes? 


30 The REL director reported that REL Mid- Atlantic’s work on school improvement plans is connected with its Mission #8 work on 
alternative school structures. 
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• Odyssey Math (Impact Study). As discussed under Mission #4, the REL director stated 
that this study was the first study to assess the impact of Odyssey Math on student 
achievement. Because Odyssey Math is a web-based, K-8 mathematics curriculum and 
assessment tool, the director noted that REL Mid-Atlantic considers it pertinent to the 
mission on use of technology in education. 

• Bridge Event on Research on Technology-Based Interventions. REL Mid-Atlantic 
proposed a Bridge Event on several studies related to the use of technology in education, 
including its own Odyssey Math impact study and IES studies on technology use. 

Quality and Relevance of Fast Response Projects 


As of December 1, 2009, 24 percent (5) of REL Mid-Atlantic’s 21 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 6-1). Twenty-five percent (3) of REL Mid-Atlantic’s 12 submissions for FRP 
reports were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 

By December 1, 2009, REL Mid-Atlantic had three FRP reports accepted and published by IES: 

1 . The Students With Disabilities Subgroup and Adequate Yearly Progress in Mid-Atlantic Region (July 
2007); 

2. Subgroups and Adequate Yearly Progress in Mid-Atlantic Region Schools (September 2007); and 

3. The Predictive Validity of Selected benchmark Assessments Used in the Mid-Atlantic Region (November 
2007). 
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Table 6-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Mid-Atlantic and all 10 RELs combined as of December 1, 2009 



Number 

Number 

Percentage 


submitted 

accepted 

accepted 

Proposals 

Mid-Atlantic 

21 

5 

24 

All RELs 

297 

137 

46 

Reports 

Mid-Atlantic 

12 

3 

25 

All RELs 

166 

92 

55 


Table Reads : Of the 21 REL Mid-Atlantic proposals submitted, 5 (24 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


Table 6-2 shows the expert panel quality and relevance ratings for these IES-published reports from 
REL Mid-Atlantic and for the three corresponding proposals. Average quality and relevance 
ratings for proposals and reports from all 10 RELs are provided for reference. Quality rating 
categories ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged 
from 1 (“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and the 
corresponding IES-published reports received a mean quality rating of 3.81. FRP proposals from 
REL Mid-Atlantic received a mean quality rating of 2.61, and the corresponding IES-published 
reports received a mean quality rating of 3.85. Among the three REL Mid-Atlantic proposals, mean 
quality ratings ranged from 2.1 1 to 2.89. Among the three REL Mid-Atlantic IES-published reports, 
mean quality ratings ranged from 3.33 to 4.38. 


31 Appendix A describes in detail the indicators that make up the quality and relevance dimensions. 
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Table 6-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Mid-Atlantic and all 10 RELs 
combined (on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All REL FRPs (N = 75) 

3.24 

3.39 

All Mid-Atlantic FRPs (N = 3) 

2.61 

2.74 

Subgroups and Adequate Yearly Progress in Mid- 
Atlantic Region Schools 

2.89 

2.33 

The Predictive Validity of Selected Benchmark 
Assessments Used in the Mid-Atlantic Region 

2.11 

2.78 

The Students With Disabilities Subgroup and Adequate 
Yearly Progress in Mid-Atlantic Region Schools 

2.83 

3.11 

Reports 

All REL FRPs (N = 91) 

3.81 

3.64 

All Mid-Atlantic FRPs (N = 3) 

3.85 

3.53 

Subgroups and Adequate Yearly Progress in Mid- 
Atlantic Region Schools 

4.38 

3.83 

The Predictive Validity of Selected Benchmark 
Assessments Used in the Mid-Atlantic Region 

3.83 

3.67 

The Students With Disabilities Subgroup and Adequate 
Yearly Progress in Mid-Atlantic Region Schools 

3.33 

3.08 


Table Reads : For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 


With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 
mean rating of 3.64 for the actual reports. REL Mid-Atlantic proposals received a mean relevance 
rating of 2.74, and the corresponding IES-published reports received a mean relevance rating of 
3.53. Among the three REL Mid-Atlantic proposals, the mean relevance ratings ranged from 2.33 to 
3.1 1. Among the three REL Mid-Atlantic IES-published reports, the mean relevance ratings ranged 
from 3.08 to 3.83. Additional detail on the distribution of expert panelist indicator-level quality and 
relevance ratings is provided in Figure 6-1. 
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Figure 6-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Mid-Atlantic 
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Figure Reads: 1.9 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on a 
5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 54 and 27 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 72 and 36 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 3 

N (Reports) = 3 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: Midwest 


7 


Regional Educational Laboratory Midwest serves a region that includes the following states: 

• Illinois; 

• Indiana; 

• Iowa; 

• Michigan; 

• Minnesota; 

• Ohio; and 

• Wisconsin. 

For the 2006-201 1 contract period, REL Midwest was housed within the research portfolio of a 
non-profit company, Learning Point Associates (LPA), headquartered in Naperville, Illinois. LPA is 
an affiliate of the American Institutes for Research (AIR). At the time of data collection, LPA had 
held previous REL contracts. 

Description of Projects and Activities 

In June 2010, a member of the evaluation study team interviewed the REL Midwest director. The 
director was asked to provide up to three examples of the work that REL Midwest had completed 
under each mission specified in ESRA. 3 ~ Other than documenting REL reports published by IES, 
the evaluation study team made no attempt to verify independently the reported information on 
REL activities, so the accuracy and completeness of this information is unknown. 

Mission #1: Training or Technical Assistance 

A component of REL Midwest’s technical assistance was its Midwest Urban Research Network 
(MURN). According to the director, this network brought together local education agencies (LEAs) 


32 For a more detailed description of the methodology, see chapter 2. 
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in the region that had between 10,000 and 60,000 students. The REL director reported that these 
districts were large enough to have their own research and evaluation offices but often did not have 
the resources to do sophisticated evaluations. The director stated that the districts came together 
once a year to network with one another and share with REL Midwest their issues and needs as they 
related to research and evaluation. MURN also allowed REL Midwest the opportunity to conduct 
Bridge Events and webinars. Bridge Events were interactive fomms conducted by the RELs that 
brought together IES-approved experts with education practitioners and policymakers. These events 
were held in-person or via webinars. At the time of the interview, REL Midwest planned to work 
with the districts in the network on how they could use their data to develop an early warning 
indicator system, meaning a system to identify students who show early warning signs that they are 
at risk for dropping out of high school. REL Midwest staff highlighted the following projects and 
activities occurring under this mission: 

• Reach and Teach. REL Midwest helped the state of Michigan develop an evaluation plan 
for the statewide initiative, Reach and Teach. To help Michigan with this evaluation, the 
REL worked with the state to develop a theory of action, which is a statement of how 
program operations are supposed to lead to desired program effects, for the initiative. Once 
the theory of action was defined, REL Midwest aided Michigan’s development of program 
measures and outcomes. At the time of the interview, the initiative was in its second year, 
and REL Midwest was still helping Michigan to shape the evaluation. 

• Indiana Full-Day Kindergarten. REL Midwest worked with Indiana to develop a plan to 
evaluate the implementation of legislation that expanded full-day kindergarten. 

• Beyond Accountability. This project aimed to help states think beyond using their data for 

accountability purposes, such as annual report cards, to answer more policy-relevant 

questions. For one component of this project, REL staff members worked with the state to 

help principals and SEAs monitor whether schools were meeting the highly qualified teacher 

33 

requirement for the Michigan Merit Curriculum. 

Mission #2: Dissemination 

To disseminate information, REL Midwest assigned staff members to each state in the region. 
Individuals in these states worked with state education agency (SEA) personnel and other 
stakeholders, such as state legislators and school board administrators. For the most recent contract, 
the REL director reported that REL Midwest made a concerted effort to visit its constituents to 
explain in detail what resources the REL could provide to the various entities. The REL also 
participated in the REL-wide dissemination effort, Ask-A-REL, through which REL Midwest 


33 http:/ / www.michigan.gov/documents/mde/New_MM C_one_pager_11.15.06_183755_7.pdf 
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constituents are able to ask the REL education-related questions. REL Midwest staff highlighted the 
following projects and activities occurring under this mission: 

• Practice Guides. In several cases, REL Midwest worked with states to disseminate 
information that the states had requested. For example, the REL director reported that REL 
Midwest worked closely with the SEA personnel in Ohio to help them better understand 
dropout issues and trends in the state. The REL used the IES practice guide on dropout 
prevention as a base of knowledge to help the state and disseminated the guide to 
participants in state meetings and working groups on dropouts. According to the director, 
based on their experiences in Ohio, REL Midwest provided this dissemination tool to other 
states in the region. 

• Bridge Events. The REL director noted that Bridge Events were one of the major tools of 
dissemination for REL Midwest. REL Midwest held events on topics such as teacher 
evaluation, assessment, and accountability. Also, the REL collaborated with other entities on 
Bridge Events, such as the Comprehensive Centers. 34 At the time of the interview, the REL 
had conducted Bridge Events on 16 topics: 

1. Interpreting Test Score Trends and Gaps 

2. Working with State Datasets: A Working Session on Methods in Data 
Handling and Report Generation: Calculating the Ability of Within-School 
Teacher Supply to Meet the Demands of New Requirements 

3. Bringing Research to Statewide Longitudinal Data Systems 

4. Evaluating Teacher Effectiveness 

5. Leading with Data: Data-Based Decision-Making and Access to Evidence- 
Based Practices; A Path to School Improvement 

6. Improving Student Achievement Through Expanded Learning 
Opportunities 

7. Calculating the Ability of Within-School Teacher Supply to Meet the 
Demands of New Requirements 

8. Improving Classroom Instruction With Formative Assessment 

9. Increasing Access to Higher Education for All Students 

10. Turning Around Chronically Low-Performing Schools 

1 1 . Educator Performance Pay: Promise and Pitfalls 

12. Successful Strategies for Effective Literacy and English Language 


34 http:/ / www2.ed.gov/programs/newccp/index.html 
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Instruction for English Language Learners in the Elementary Grades 

13. Transforming Struggling Schools 

14. Effective Leadership to Transform Struggling Schools 

15. Online Learning Opportunities for Rural Schools 

16. Increasing Graduation Rates by Creating a System of Supports 

• Participation in Conferences. Another REL Midwest means of dissemination was 
participation in conferences with targeted audiences and topics. This participation often 
included presentations on the REL program and other resources available from IES. 

Mission #3: Needs Assessment or Needs Sensing 

REL Midwest staff highlighted the following projects and activities occurring under this mission: 

• Formal Needs Assessment. REL Midwest had an outside research firm, Public Agenda, 
conduct formal regional needs assessments annually. The formal needs assessment was 
completed using telephone surveys of superintendents, principals, teachers, and school 
board members across all seven states in the region. REL Midwest also conducted focus 
groups with these constituents to better understand how the REL could meet their needs. 

To involve the broader public, including parents of students, REL Midwest held town hall 
meetings to address emerging issues in education and to get the public’s feedback on ways 
the REL could help. 

• Technical Assistance Outreach and Dissemination. REL Midwest assigned personnel to 
each state in the region who worked with key stakeholders, including SEA personnel, state 
legislators, school board administrators, and other stakeholders. The director stated that, at 
meetings with stakeholders, REL Midwest staff members asked their constituents how the 
REL could best meet their needs. Also, during technical assistance outreach efforts, REL 
Midwest staff members inquired about constituents’ needs. 

• Request and Response Database. REL Midwest logged constituents’ requests and the 
lab’s responses to these inquiries in the lab’s Request and Response database. This log also 
helped the REL evaluate the needs of the region. 

Mission #4: Carrying Out Applied Research Projects 

At the time of the interview, the REL was in the process of completing two impact studies. Both 
studies were 2 years in duration, included approximately 30 schools each, and were due to be 
completed in December 2010. REL Midwest staff highlighted the following projects and activities 
occurring under this mission: 

• The Impact of the Measures of Academic Progress on Differentiated Instruction and 
Student Achievement (Impact Study). This randomized study examined the impact of a 
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professional development program designed and delivered by the Northwest Evaluation 
Association 35 (NWEA) in elementary schools. The program. Measures of Academic Progress 
(MAP), is designed to increase teacher efficacy. This evaluation explored whether this 
assumption was tme based on student and teacher outcome measures. 

• Adolescent Literacy (Impact Study). This randomized study evaluated the effectiveness 
of the Content Literacy Continuum (CLC) intervention in high schools. Specifically, the study 
looked at the extent to which the CLC intervention improved students’ reading skills, 
content knowledge, and other academic outcomes. In addition, the study examined the 
effect of this literacy approach on the quality of literacy instruction by language arts teachers 
and teachers of other subjects. The study also explored the factors that promoted or 
impeded the successful implementation of the intervention in ninth-grade classrooms. 

Mission #5: Providing Applied Research Projects in a Usable Format 

REL Midwest reported that while it used Fast Response Projects (FRPs) as a source for providing its 
constituents with applied research projects in usable formats, it also used other mechanisms as 
means of disseminating this information. For example, REL Midwest used Policy Briefs and Bridge 
Events to connect research to its constituents. The REL director noted that Policy Briefs were 
typically no more than two pages long. The director stated that they were shorter than other 
documents because the REL recognized that the audience that read Policy Briefs (e.g., legislators) 
was different from the audience that read impact evaluations (e.g., researchers). The director 
reported that REL Midwest saw Bridge Events as an important tool in communicating research 
projects to its constituency. REL Midwest staff highlighted the following projects and activities 
occurring under this mission: 


• Bridge Event on Achievement Gaps. This was an interactive event in which school 
district participants used Excel to view their district’s achievement scores. Participants 
downloaded their respective district’s data and ran preloaded macros, based on different 
assumptions, to view the achievement data. Achievement gap data were also presented in 
graphs. 

• Bridge Event on Formative Assessment. This event featured experts who presented 
information to school district participants on the role of formative assessment in a 
traditional panel setting. REL Midwest reported that about 1 80 people attended the event. 
According to the REL director, while the participants found the event valuable, they 
requested that REL Midwest do a follow-up event that explains to district personnel how to 
teach people in their district to implement formative assessment. At the time of the 
interview, the director said that REL Midwest was in the process of seeking permission from 
IES to develop an interactive video for districts to use. In addition to distributing the video. 


35 http://www.nwea.org/ 
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REL Midwest hoped to also make podcasts and provide the video for download on its 
website. 

• Bridge Event on After-School Programs. REL Midwest held a Bridge Event on after- 
school programs to which it invited experts on the topic to address school district 
participants. The REL also conducted a follow-up to the event with an emphasis on 
involving teachers in the discussion. It videotaped teachers discussing how they 
implemented these programs in their schools. This video was available for download on 
REL Midwest’s website. 

Mission #6: Coordination and Collaboration 

REL Midwest worked with other RELs as well as other ED-funded entities, such as the 
Comprehensive Centers, to write reports, conduct Bridge Events, and provide technical assistance. 
In addition, the REL participated in cross-REL working groups, such as the Rural Working Group, 
the American Recovery and Reinvestment Act (ARRA) Working Group, and the Data Working 
Group. REL Midwest also participated in cross-REL collaboration via Thursdays @ 3, the REL 
director meetings, the National Laboratory Network Intranet, and the National Reference Desk. 
REL Midwest staff highlighted the following projects and activities occurring under this mission: 


• Native American Education Report (FRP). REL Midwest participated in a multi-REL 
report on Native American educational achievement. This report was led by REL 
Northwest. 

• Collaboration With Other ED-Funded Organizations. REL Midwest collaborated with 
organizations such as the Comprehensive Centers, the Center for Teacher Quality 36 , and the 
Center for Educator Compensation Reform (CECR). 3 In these collaborations, the REL 
worked with the other organizations to conduct Bridge Events, Research to Action 
Forums, 38 and technical assistance projects. For instance, when the Comprehensive Centers 
received a request that they did not have the resources and/ or expertise to answer, they 
sometimes approached the REL. The REL director said that, in some cases, the REL 
answered the request by writing a short brief or research review. 

• Collaboration With Other RELs. At the time of the interview, the REL was working with 
other RELs on a project about teacher evaluation. The REL director reported that in light of 
Race to the Top, 39 this was a popular topic. The REL anticipated that this project would 
result in a written product accompanied by a Bridge Event. The REL was working on two 
projects that were bringing all 10 RELs together. One such project focused on access to 
higher education in rural communities. 


36 http:/ / teachingquality.org/ 

37 http://cecr.ed.gov/ 

38 http: / / researchtoactionforum.org/ 

39 http:/ /www2.ed.gov/programs/racetothetop-assessment/index.html 
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Mission #7: School Finance Systems 

REL Midwest staff highlighted the following projects and activities occurring under this mission: 

• Bridge Event on Teacher Compensation. In partnership with the CECR, REL Midwest 
completed a 2-day Bridge Event on teacher compensation, an event that highlighted the 
different approaches that school districts and states are using for teacher compensation 
models. 

• Project on School Funding in Minnesota. At the time of the interview, and pending IES 
acceptance of the project, REL Midwest planned to work on an exploratory analysis of 
school funding in Minnesota, with particular emphasis on the similarities and differences 
between rural and urban districts. 4 " 

• Study on Illinois’ Pension System. REL Midwest planned to do a study on the Illinois 
pension system for public educators. However, the director reported that the REL was 
having difficulty obtaining the necessary data. 

Mission #8: Alternative Administrative Structures 

REL Midwest staff highlighted the following projects and activities occurring under this mission: 

• Bridge Event on Leadership. REL Midwest held a Bridge Event in Indiana on different 
administrative models and effective leaders. An invited expert led the event. 

• Project on Administrative Costs. Several states, including Iowa and Ohio, requested that 
REL Midwest help them better understand the various options available to districts to cut 
administrative costs, for example, by sharing busing costs. At the time of the interview, the 
director reported that this project was in its beginning stages. 

Mission #9: School Improvement Plans 

REL Midwest staff highlighted the following project or activity occurring under this mission: 

• School Improvement Plans ( FRF ). At the time of the interview, REL Midwest was 
working on a descriptive report that reviews school districts’ school improvement plans in 
six of the states in the region. 

Mission #10: Technology in Education 

REL Midwest staff highlighted the following project or activity occurring under this mission: 

• Bridge Event on Distance Education in Rural Communities. At the time of the 
interview, the REL was planning to hold a Bridge Event on distance education in rural 
communities. This event was to focus on the use of distance education in different mral 
communities. The event addressed such issues as different models of distance education, 


40 IES published a report from this study in February 2012. 
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how different communities are approaching distance education, and ways to ensure that 
teachers are qualified to teach distance education courses. 

Quality and Relevance of Fast Response Projects 


As of December 1, 2009, 39 percent (11) of REL Midwest’s 28 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 7-1). Fifty percent (7) of REL Midwest’s 14 submissions for FRP reports 
were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 


By December 1, 2009, REL Midwest had seven FRP reports accepted and published by IES: 

1 . Getting the Evidence for Evidence-Based Initiatives: Eloiv the Midwest States Use Data Systems to Improve 
Education Processes and Outcomes (June 2007); 

2. Improving Survey Research On Two-Year College Experiences (August 2007); 

3. Examining District Guidance to Schools on Teacher Evaluation Policies in the Midwest Region 
(November 2007); 

4. Calculating the Ability ofWithin-School Teacher Supply to Meet the Demands of New Requirements: The 
Example of the Michigan Merit Curriculum (July 2008); 


Table 7-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Midwest and all 10 RELs combined as of December 1, 2009 



Number 

Number 

Percentage 


submitted 

accepted 

accepted 

Proposals 

Midwest 

28 

11 

39 

All RELs 

297 

137 

46 

Reports 

Midwest 

14 

7 

50 

All RELs 

166 

92 

55 

Table Reads: Of the 28 REL Midwest proposals submitted, 11 (39 percent) were accepted. 


SOURCE: U.S. Department of Education, NCEERA. 


Table 7-2 shows the expert panel quality and relevance ratings for these IES-published reports from 
REL Midwest and for the six corresponding proposals. 41 Average quality and relevance ratings for 
proposals and reports from all 10 RELs are provided for reference. Quality rating categories ranged 


41 One report lacked a corresponding proposal, as noted in Table 7-2. Appendix A describes in detail the indicators that make up the 
quality and relevance dimensions. 
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from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged from 1 (“not 
relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

Table 7-2. Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Midwest and all 10 RELs combined 
(on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All REL FRPs (N = 75) 

3.24 

3.39 

All Midwest FRPs (N = 6) 

3.41 

3.54 

Examining District Guidance to Schools on Teacher 
Evaluation Policies in the Midwest Region 

3.11 

3.78 

Getting the Evidence for Evidence-Based Initiatives: 
How the Midwest States Use Data Systems to Improve 
Education Processes and Outcomes 

2.72 

2.78 

Improving Survey Research on Two-Year College 
Experiences 

3.94 

3.89 

Methodologies Used by Midwest Region States for 
Studying Teacher Supply and Demand 

3.56 

3.67 

Snapshots of Indiana’s Full-Day Kindergarten Programs 
Before and After the State’s Funding Increase for the 
Program 

4.00 

3.33 

State Policies on Teacher Evaluation Practices in the 
Midwest Region 

3.11 

3.78 
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Table 7-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Midwest and all 10 RELs combined 
(on a 5-point scale with 5 being the highest) (continued) 


Mean Ratings 

Quality 

Relevance 

Reports 


All REL FRPs (N = 91) 

3.81 

3.64 

All Midwest FRPs (N = 7) 

3.55 

3.36 

Calculating the Ability of Within-School Teacher Supply 
to Meet the Demands of New Requirements: The 
Example of the Michigan Merit Curriculum(-) 

4.10 

3.44 

Examining District Guidance to Schools on Teacher 
Evaluation Policies in the Midwest Region 

4.13 

4.00 

Getting the Evidence for Evidence-Based Initiatives: 
How the Midwest States Use Data Systems to Improve 
Education Processes and Outcomes 

2.67 

2.58 

Improving Survey Research On Two-Year College 
Experiences 

2.78 

3.33 

Methodologies Used by Midwest Region States for 
Studying Teacher Supply and Demand 

3.92 

3.75 

Snapshots of Indiana’s Full-Day Kindergarten 
Programs Before and After the State’s Funding 
Increase for the Program 

3.75 

3.00 

State Policies on Teacher Evaluation Practices in the 
Midwest Region 

3.50 

3.42 


Table Reads: For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. (-) = A proposal was not available for the report, “Calculating the Ability of Within-School Teacher Supply to Meet the Demands of 
New Requirements: The Example of the Michigan Merit Curriculum." 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 


For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and IES-published 
reports received a mean quality rating of 3.81. FRP proposals from REL Midwest received a mean 
quality rating of 3.41, and the corresponding IES-published reports received a mean quality rating of 
3.54. Among the six REL Midwest proposals, mean quality ratings ranged from 2.72 to 4.00. Among 
the seven REL Midwest IES-published reports, mean quality ratings ranged from 2.67 to 4.13. 
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With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 
mean rating of 3.64 for the actual reports. REL Midwest proposals received a mean relevance rating 
of 3.54, and the corresponding IES-published reports received a mean relevance rating of 3.36. 
Among the six REL Midwest proposals, the mean relevance ratings ranged from 2.78 to 3.89. 
Among the seven REL Midwest IES-published reports, the mean relevance ratings ranged from 2.58 
to 4.00. Additional detail on the distribution of expert panelist indicator-level quality and relevance 
ratings is provided in Figure 7-1 . 


Figure 7-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Midwest 
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Figure Reads: 12.0 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on 
a 5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 108 and 54 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 161 and 81 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 6 

N (Reports) = 7 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: 

Northeast and Islands 



Regional Educational Laboratory Northeast and Islands serves a region that includes the following 
jurisdictions: 


• Connecticut; 

• Maine; 

• Massachusetts; 

• New Hampshire; 

• New York; 

• Puerto Rico; 

• Rhode Island; 

• U.S. Virgin Islands; and 

• Vermont. 


For the 2006-201 1 contract period, REL Northeast and Islands was based in the Research 
Evaluation and Policy unit in the Domestic Education (a.k.a. Learning and Teaching) division at the 
Education Development Center (EDC) in Newton, Massachusetts. The main subcontractors to 
EDC were Learning Innovations at WestEd and the American Institutes for Research (AIR). 
Learning Innovations worked primarily on Fast Response Projects (FRPs) and dissemination of 
materials. AIR worked on impact studies. Although EDC held one of the original REL contracts, it 
did not hold the previous REL contract. Therefore, the organization was considered a new REL for 
the 2006-201 1 contract period. 


Description of Projects and Activities 


In August 2010, a member of the evaluation study team interviewed the REL Northeast and Islands 
director and five staff members. REL staff members were asked to provide up to three examples of 
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the work their REL had completed under each mission specified in ESRA. 42 Other than 
documenting REL reports published by IES, the evaluation study team made no attempt to verify 
independently the reported information on REL activities, so the accuracy and completeness of this 
information is unknown. 

Mission #1: Training or Technical Assistance 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 
this mission: 

• Policy Challenge Conferences. The REL held two conferences — one focused on special 
populations of students, such as at-risk students and English language learners (ELLs), and 
the other focused on teacher quality issues. The REL brought together nine people from 
each of its jurisdictions to attend each of the conferences. In addition to the REL’s 
attendees, other participants such as the Comprehensive Centers were invited to attend. 
Keynote speakers who were experts in the focus areas addressed the participants. 

• Low-Performing Schools Webinar Series. The REL conducted a webinar based on an 
FRP that REL Northeast and Islands published on low-performing schools. Following the 
webinar, one of the associate commissioners from the New York Department of Education 
(NYDOE) contacted REL Northeast and Islands and requested two additional webinars: (1) 
one that included the report authors, NYDOE, and other states that had done similar work, 
and (2) a second that included NYDOE and other SEAs in the region (such as Connecticut, 
Massachusetts, and Rhode Island) to learn what those SEAs have been doing to help low- 
performing schools. In addition, REL Northeast and Islands led a similar event in Puerto 
Rico. 

• Bridge Events. REL Northeast and Islands held a Bridge Event in the form of a regional 
webinar on the Dropout Prevention Practice Guide. The REL reported that over 100 people 
attended the event, during which an expert in the field presented and provided an 
opportunity for questions from the attendees. Following the Bridge Event, the 
commissioner of the Virgin Islands requested that REL Northeast and Islands present this 
information at a meeting in the Virgin Islands to the governor, legislators, and ED 
representatives, followed by separate presentations to each of the school districts in the 
Virgin Islands. In addition to the Bridge Event described above, REL Northeast and 
Islands, at the time of the interview, had held Bridge Events on seven additional topics: 

1 . Effective Literacy and English Language Instruction for English Learners in the 
Elementary Grades 

2. State Investments in Early Childhood Education Programs 

3. Reducing Behavior Problems in the Elementary School Classroom 


42 For a more detailed description of the methodology, see chapter 2. 
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4. Implications of the Final Report of the National Mathematics Advisory Panel 

5. Increasing Access to Higher Education for All Students 

6. Dropout Prevention: Keeping Our Students in School 

7. Implementing Response to Intervention (Rtl) in Mathematics: Research-Based 
Strategies for Maine, New Hampshire, and Vermont Grades K-8 Schools 

Mission #2: Dissemination 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 
this mission: 

• Website. REL Northeast and Islands’ website was launched in October 2008. According to 
the REL, the website had roughly 3,000 unique visitors each month and provided access to 
all of REL Northeast and Islands’ work, including reports, newsletters, and webinars. The 
website housed a page called the “Reference Desk Digest” that provided a compilation of 
Reference Desk responses on a similar topic that, according to the REL, receives more 
traffic than any other page on the site. The REL Northeast and Islands team also reported 
that the website referred more people to the IES REL site than any other website. During 
the interview, the team stated that it was redesigning the site and planned to implement the 
changes by the end of summer 2010. 

• Newsletters. REL Northeast and Islands produced and disseminated several newsletters 
and publications. The first example was ED Evidence , which was sent quarterly via email to 
the REL’s constituents. This newsletter documented any REL-sponsored events that 
occurred within a specified timeframe, named recently published REL reports, and 
contained questions and answers with an IES-approved expert. A second example was ED 
Evidence Issues, which was distributed by email to the REL’s constituents bimonthly. This 
newsletter focused on a specific education issue and documented all of REL Northeast and 
Islands work around the topic, as well as all IES publications around the issue. A final 
example was Monthly Email Update, which was a newsletter for constituents that documented 
all new content on the REL’s website, including the links to the content. 

• Webinars. REL Northeast and Islands disseminated information to its constituency via 
webinars. The REL tracked who signed up for and attended the webinars. To show the 
location of the participants, the REL produced a map that highlighted the various cities and 
towns from which people dialed in to participate in the webinars. In addition to traditional 
dissemination webinars, the REL created some bridge webinars in which a smaller group 
communicated by phone during the webinar. Finally, the REL made the PowerPoint slides 
as well as an audio archive from the webinars available to people who were unable to attend 
the sessions. 


Evaluation of the Regional Educational Laboratories: Interim Report 

71 


Mission #3: Needs Assessment or Needs Sensing 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 
this mission: 

• Governing Board. REL Northeast and Islands’ Governing Board comprised 27 members 
and met three times a year. During the meetings, the REL presented the work that it had 
done and asked the Governing Board for feedback and direction. In addition, the REL 
invited practitioners to speak in front of the Board to illustrate how they had used REL 
research. A keynote speaker also addressed the Board and discussed an issue selected by the 
Board in advance of the meeting. The REL director reported that outside of the Board 
meetings, Governing Board members participated in other REL activities, met with state 
liaisons/researchers, and played an important role in shaping the Updated Annual Plans 
(UAPs). 

• State Liaisons. Each state or jurisdiction under REL Northeast and Islands was assigned a 
two-member team from the REL. Each of these teams was made up of a researcher and one 
person with extensive knowledge of the respective state to which s/he was assigned. The 
person with the extensive state knowledge was responsible for facilitating conversations with 
the state about ways in which the REL could help, while the researcher considered ways to 
take the issues the state raised and turn them into research projects for the REL. The REL 
stated that these teams were not housed within SEAs, so they were able to sense stakeholder 
needs outside of the SEA as well as inside it. 

• Conferences. REL staff members attended conferences and meetings, such as curriculum 
association or superintendents’ meetings, to network in the region and conduct needs 
sensing through conversations with constituents. At conferences, the REL would sometimes 
have a booth for conference participants to obtain information and ask questions. At these 
events, the REL spoke with participants and generated questions for the Reference Desk. 
According to the REL team, by the time of the interview, it had received more than 50 
Reference Desk requests from conference participants. 

Mission #4: Carrying Out Applied Research Projects 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 
this mission: 

• Thinking Reader (Impact Study). This study was designed to examine the impact of an 
interactive computer-based program (Thinking Reader) on reading achievement. Thinking 
Reader, a program designed to help middle school students who are struggling with reading 
comprehension, uses books in digitized forms with built-in supports. The study was a 
randomized study of sixth-grade students and approximately 90 teachers in 35 elementary 
and middle schools. Control schools received the program and training following the study. 
The first draft of this study was submitted to IES in December 2009. 43 At the time of the 


43 IES published a report from this study in April 2011. 


Evaluation of the Regional Educational Laboratories: Interim Report 

72 


interview, REL Northeast and Islands planned to extend this report to add exploratory 
chapters on subgroup analyses. 


• Virtual Algebra (Impact Study). The Virtual Algebra study examined whether offering 
algebra I as an online course was a viable option for enhancing access to this coursework by 
examining differences in the later course-taking patterns of eighth-grade students who did 
and did not take online algebra I courses. According to the REL, some policymakers had 
argued that students should have access to a full algebra I course in the eighth grade because 
it is an important gateway course that enables students to be successful in their future 
education endeavors. However, the REL team stated that some students did not have access 
to algebra I in the eighth grade, particularly mral students. The study was conducted in 72 
mral schools in Maine and Vermont and tracked students in these schools to see what 
courses they took in ninth grade and tenth grades. Originally there were 74 schools in the 
study, but two schools dropped out. At the time of the interview, the REL team said that the 
report would be submitted for review in early fall 201 0 44 . 

Mission #5: Providing Applied Research Projects in a Usable Format 

The REL reported it had published approximately 20 FRPs as of August 2010 and had two Fast 
Response Projects in the final editing stages. REL Northeast and Islands staff highlighted the 
following projects and activities occurring under this mission: 

• Study on Mathematics Instruction and Special Education (FRP). REL Northeast and 
Islands wrote an FRP on integrating special education students into mathematics instruction 
and identified recommended practices in mathematics instruction for special education 
students. This report was qualitative and involved case studies at the school level. 

• Study on ELL Performance in Large-Scale Assessments (FRP). Three states in REL 
Northeast and Islands’ region shared a common state assessment, including an English 
language learner (ELL) assessment. These states were interested in exploring the relationship 
between the two assessments. This FRP provided an examination of how ELL student 
performance on the state assessment related to student competency scores on the ELL 
assessment, such as written fluency, speaking fluency, comprehension fluency, and listening 
fluency. 

• Study on Who Reports Bullying (FRP). Using a data set of self-reported student survey 
data from the U.S. Department of Health and Human Services, this report examined the 
types of bullying students encounter in school and whether the bullying was reported to the 
authorities. The study also examined if student-level or school-level data predicted whether 
bullying was reported to the authorities. The REL team noted that this report was requested 
by one of the REL’s Governing Board members. At the time of the interview, the REL said 
that the report would be released in the near future. 41 


44 IES published a report from this study in December 2011. 

45 IES published a report from this study in August 2010. 
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Mission #6: Coordination and Collaboration 

REL Northeast and Islands collaborated with several different types of organizations, including the 
state departments of education, school districts, federally funded technical assistance providers, 
federal Comprehensive Centers, other RELs, and offices of the U.S. Department of Education 
(ED). The REL director mentioned that REL Northeast and Islands worked with ED and the 
Comprehensive Centers to convene an event about the American Recovery and Reinvestment Act 
(ARRA) in which ED representatives spoke to the states. REL Northeast and Islands staff 
highlighted the following projects and activities occurring under this mission: 

• Bridge Events. The REL team reported that REL Northeast and Islands partnered with 
another organization on 14 out of 15 of the Bridge Events it held. The REL often partnered 
with the stakeholder organizations for which it was providing the events to make them feel 
as if the event was really theirs. These partnerships helped the REL with recmitment for the 
events. 

• Fast Response Projects. REL Northeast and Islands collaborated with various RELs on 
FRPs on the following topics: Native American education, State Fiscal Stabilization Fund 
(teacher effectiveness), and student dropouts. For the work on student dropouts, REL Mid- 
Atlantic approached REL Northeast and Islands to ask for help in replicating REL 
Northeast and Islands’ report and database on dropout prevention programs. At the time of 
the interview, REL Northeast and Islands planned to merge the datasets created by both 
RELs to form one database on dropout prevention programs for both regions. 

• National Laboratory Network Working Groups. According to the REL team, REL 
Northeast and Islands participated in all REL working groups and was involved in starting 
some of the working groups. In addition to participating in the working groups, the REL 
participated in the National Laboratory Network’s Thursdays @ 3, hosting approximately six 
of these events by the time of the interview. 

Mission #7: School Finance Systems 46 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 
this mission: 

• Reference Desk Responses. One of REL Northeast and Islands’ Governing Board 
members, who was also a district superintendent, asked the REL to examine how school 
districts that are struggling financially can ensure they can provide students with a solid 
education. The REL responded by developing a series of Reference Desk questions based 
on this issue. Examples of questions the REL considered include: (1) What does the research 
say about organizing and managing district finances to maximize the use of limited funds? 

(2) What does the literature say about district-wide costs that may unnecessarily add to total 


46 REL Northeast and Islands described three activities under this mission that are closely related to the activities described under 
Mission #8, Alternative Administrative Structures. 
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district expenses? At the time of the interview, REL Northeast and Islands indicated that it 
hoped to write an FRP on this subject. 

• Fast Response Project on Costs of Teacher Turnover. At the time of the interview, REL 
Northeast and Islands planned to submit a proposal in fall 2010 based on the following 
research question: What is the cost of teacher turnover? To write this report, the REL 
planned to work closely with financial personnel in two school districts. 

• Reference Desk Responses. REL Northeast and Islands responded to Reference Desk 
questions on several related issues, such as how cost effectiveness relates to re-districting for 
the optimal district size and how cost effectiveness relates to school size. 

Mission #8: Alternative Administrative Structures 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 
this mission: 

• Reference Desk Responses. As was described under Mission #7, REL Northeast and 
Islands planned to conduct research on how school districts that are struggling financially 
can ensure they can still provide students with a quality education. 

• Fast Response Project on Cost of Teacher Turnover. As was discussed under Mission 
#7, REL Northeast and Islands planned to submit a proposal in fall 2010 based on the 
following research question: What is the cost of teacher turnover? 

• Reference Desk Responses. The REL team responded to several Reference Desk 
questions on issues such as charter schools, alternative schools for students with behavior 
problems, emancipated migrant youth, and freshman academics, all pertinent to alternative 
administrative structures. 

Mission #9: School Improvement Plans 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 
this mission: 

• Bridge Event on Response to Intervention (Rtl j — Mathematics. The REL presented a 
Bridge Event webinar on the Rtl in mathematics practice guide. The event was geared to the 
northern tier region (New Hampshire, Vermont, and Maine) of REL Northeast and Islands 
because, as reported by the REL director, many schools in these states have a limited 
number of highly qualified math teachers who are able to implement Rtl strategies in their 
classrooms. Other states and districts then asked the REL to do a similar event for them. 

The REL noted that for these additional events, the practice guide author sometimes worked 
via webinar with specific districts on ways they could most effectively implement strategies; 
in other instances, the REL conducted the additional event in person. The REL director 
reported that Rtl was important to many districts because it was a component in their school 
improvement plans. 
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• Bridge Events on Teacher Effectiveness. REL Northeast and Islands worked with the 
New England Comprehensive Center on Bridge Events and technical assistance projects 
around issues of teacher quality. Specifically, the REL convened several meetings with state 
department leaders for the New England Collaborative for Educational Quality on issues 
related to teacher effectiveness. The events focused on directives around teacher 
effectiveness and addressed these questions: (1) How can states work out better reciprocity 
agreements for teacher certification? (2) What can states learn from one another about 
teacher induction programs? 

Mission #10: Technology in Education 

REL Northeast and Islands staff highlighted the following projects and activities occurring under 

this mission: 

• Virtual Algebra (Impact Study). Described in more detail under Mission #4, the Virtual 
Algebra study examined whether offering algebra I as an online course was a viable option to 
provide access to this coursework by examining differences in the course-taking patterns of 
eighth-grade students who did and did not take online algebra I courses. 

• Thinking Reader (Impact Study). Described in more detail under Mission #4, this study 
was designed to evaluate the strategies supported by Thinking Reader, a program to help 
middle school students who are stmggling with reading comprehension by using books in 
digitized forms with built-in supports to help them. 

• Technical Assistance. At the time of the interview, in response to school districts’ 
questions related to online algebra courses, the REL planned to write a report outlining the 
different on-line algebra courses available. This descriptive rather than evaluative work 
would answer questions such as: (1) What is the response time from teacher to student? (2) 

Is there a discussion board? (3) Does the course allow for instantaneous feedback? 

Quality and Relevance of Fast Response Projects 


As of December 1, 2009, 56 percent (25) of REL Northeast and Islands’ 45 FRP proposals were 
accepted by NCEERA and approved to continue as IES-supported projects, compared to 46 
percent for all 10 RELs combined (Table 8-1). Sixty-three percent (17) of REL Northeast and 
Islands’ 27 submissions for FRP reports were accepted for publication by IES, compared to 55 
percent for all 1 0 RELs combined. 
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Table 8-1. 


Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Northeast and Islands and all 10 RELs combined as of December 1 , 2009 



Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 

Northeast and Islands 

45 

25 

56 

All RELs 

297 

137 

46 

Reports 

Northeast and Islands 

27 

17 

63 

All RELs 

166 

92 

55 


Table Reads : Of the 45 REL Northeast and Islands proposals submitted, 25 (56 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


By December 1, 2009, REL Northeast and Islands had 17 FRP reports accepted and published by 
IES: 

1 . Measuring How Benchmark Assessments Affect Student Achievement (December 2007); 

2. A Description of Foundation Skills Interventions for Struggling Middle-Grade Readers in Four Urban 
Northeast and Islands Region School Districts (February 2008); 

3. Piloting a Searchable Database of Dropout Prevention Programs in Nine Low-Income Urban School 
Districts in the Northeast and Islands Legion (March 2008); 

4. An Analysis of State Data on the Distribution of Teaching Assignments Filled by Highly Qualified 
Teachers in New York Schools (April 2008); 

5. A Second Follow-up Year for Measuring How Benchmark Assessments Affect Student Achievement 
(April 2008); 

6. Gender Gaps in Assessment Outcomes in Vermont and the United States (August 2008); 

7. Math Education Practices for Students With Disabilities and Other Struggling Learners: Case Studies of 
Six Schools in Two Northeast and Islands Region States (August 2008); 

8. Performance Patterns for Students With Disabilities in Grade 4 Mathematics Education in New York 
State (August 2008); 

9. Performance Patterns for Students With Disabilities in Grade 4 Mathematics Education in Massachusetts 
(August 2008); 

1 0. Developing the Compendium of Strategies to Leduce Teacher Turnover in the Northeast and Islands Legion: 
A Companion to the Database (October 2008); 

1 1 . New Measures of English Language Proficiemy and Their Relationship to Performance on Large-Scale 
Content Assessments (January 2009); 
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12. How Eight State Education Agencies in the Northeast and Islands Region Identify and Support Eow- 
Pe forming Schools and Districts (March 2009); 

13. Five States’ Efforts to Improve Adolescent Eiteracy (April 2009); 

14. Parent Involvement Strategies in Urban Middle and High Schools in the Northeast and Islands Region 
(April 2009); 

15. How State Education Agencies in the Northeast and Islands Region Support Data-Driven Decision 
Making in Districts and Schools (May 2009); 

16. Analysing Performance by Grade 10 Hispanic High School Students on the Massachusetts State 
Assessment (June 2009); and 

17. Features of State Response to Intervention Initiatives in Northeast and Islands Region States (November 
2009). 

Table 8-2 shows the expert panel quality and relevance ratings for these IES-published reports from 
REL Northeast and Islands and for the 15 corresponding proposals. 47 Average quality and relevance 
ratings for proposals and reports from all 10 RELs are provided for reference. Quality rating 
categories ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged 
from 1 (“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and IES-published 
reports received a mean quality rating of 3.81. FRP proposals from REL Northeast and Islands 
received a mean quality rating of 3.25, and the corresponding IES-published reports received a mean 
quality rating of 3.96. Among the 15 REL Northeast and Islands proposals, mean quality ratings 
ranged from 2.17 to 4.24. Among the 17 individual REL Northeast and Islands IES-published 
reports, mean quality ratings ranged from 2.88 to 4.58. 


47 One proposal resulted in two reports, as noted in Table 8-2. Appendix A describes in detail the indicators that make up the quality 
and relevance dimensions. 
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Table 8-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Northeast & Islands and all 10 RELs 
combined (on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All RELFRPs (N = 75) 

3.24 

3.39 

All Northeast & Islands FRPs ( N = 15) 

3.25 

3.56 

A Description of Foundation Skills Interventions for Struggling 
Middle-Grade Readers in Four Urban Northeast and Islands Region 
School Districts 

2.59 

3.44 

An Analysis of State Data on the Distribution of Teaching 
Assignments Filled by Highly Qualified Teachers in New York Schools 

2.35 

3.12 

Analyzing Performance by Grade 10 Hispanic High School Students 
on the Massachusetts State Assessment 

4.24 

4.33 

Developing the Compendium of Strategies to Reduce Teacher 
Turnover in the Northeast and Islands Region: A Companion to the 
Database 

2.71 

3.22 

Features of State Response to Intervention Initiatives in Northeast 
and Islands Region States 

3.50 

3.56 

Five States’ Efforts to Improve Adolescent Literacy 

3.39 

3.67 

Gender Gaps in Assessment Outcomes in Vermont and the United 
States 

3.83 

3.89 

How Eight State Education Agencies in the Northeast and Islands 
Region Identify and Support Low-Performing Schools and Districts 

3.72 

3.78 

How State Education Agencies in the Northeast and Islands Region 
Support Data-Driven Decision Making in Districts and Schools 

3.28 

3.78 

Math Education Practices for Students With Disabilities and Other 
Struggling Learners: Case Studies of Six Schools in Two Northeast 
and Islands Region States 

3.17 

3.78 

Measuring How Benchmark Assessments Affect Student 
Achievement [and Second Follow-up Year] 

3.56 

3.22 

New Measures of English Language Proficiency and Their 
Relationship to Performance on Large-Scale Content 

3.83 

3.56 

Parent Involvement Strategies in Urban Middle and High Schools in 
the Northeast and Islands Region 

3.67 

3.56 

Performance Patterns for Students With Disabilities in Grade 4 
Mathematics Education in Massachusetts [and New York State] 

2.78 

3.56 
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Table 8-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Northeast & Islands and all 10 RELs 
combined (on a 5-point scale with 5 being the highest) (continued) 


Mean Ratings 

Quality 

Relevance 

Piloting a Searchable Database of Dropout Prevention Programs in 
Nine Low-Income Urban School Districts in the Northeast and 
Islands Region 

2.17 

2.89 

Reports 


All REL FRPs (N = 91) 

3.81 

3.64 

All Northeast & Islands FRPs (N = 17) 

3.96 

3.70 

A Description of Foundation Skills Interventions for Struggling 
Middle Grade Readers in Four Urban Northeast and Islands Region 
School Districts 

2.88 

3.25 

A Second Follow-up Year for Measuring How Benchmark 
Assessments Affect Student Achievement 

3.94 

3.38 

An Analysis of State Data on the Distribution of Teaching 
Assignments Filled by Highly Qualified Teachers in New York 
Schools 

4.35 

4.00 

Analyzing Performance by Grade 10 Hispanic High School Students 
on the Massachusetts State Assessment 

4.50 

3.92 

Developing the Compendium of Strategies to Reduce Teacher 
Turnover in the Northeast and Islands Region: A Companion to the 
Database 

3.50 

3.58 

Features of State Response to Intervention Initiatives in Northeast 
and Islands Region States 

3.96 

3.67 

Five States’ Efforts to Improve Adolescent Literacy 

3.71 

4.08 

Gender Gaps in Assessment Outcomes in Vermont and the United 
States 

4.04 

3.17 

How Eight State Education Agencies in the Northeast and Islands 
Region Identify and Support Low-Performing Schools and Districts 

4.58 

3.92 

How State Education Agencies in the Northeast and Islands Region 
Support Data-Driven Decision Making in Districts and Schools 

3.67 

3.75 

Math Education Practices for Students With Disabilities and Other 
Struggling Learners: Case Studies of Six Schools in Two Northeast 
and Islands Region States 

4.25 

3.92 

Measuring How Benchmark Assessments Affect Student 
Achievement 

3.94 

3.38 

New Measures of English Language Proficiency and Their 
Relationship to Performance on Large-Scale Content 

3.62 

3.33 

Parent Involvement Strategies in Urban Middle and High Schools in 
the Northeast and Islands Region 

4.46 

4.17 

Performance Patterns for Students With Disabilities in Grade 4 
Mathematics Education in Massachusetts 

4.27 

3.88 

Performance Patterns for Students With Disabilities in Grade 4 
Mathematics Education in New York State 

4.27 

3.88 
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Table 8-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Northeast & Islands and all 10 RELs 
combined (on a 5-point scale with 5 being the highest) (continued) 


Mean Ratings 

Quality 

Relevance 

Piloting a Searchable Database of Dropout Prevention Programs in 
Nine Low-Income Urban School Districts in the Northeast and 

Islands Region 3.43 

3.67 


Table Reads : For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 

NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 


With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 
mean rating of 3.64 for the actual reports. REL Northeast and Islands proposals received a mean 
relevance rating of 3.56, and the corresponding IES-published reports received a mean relevance 
rating of 3.70. Among the 15 REL Northeast and Islands proposals, the mean relevance ratings 
ranged from 2.89 to 4.33. Among the 17 REL Northeast and Islands IES-published reports, the 
relevance ratings ranged from 3.17 to 4.17. Additional detail on the distribution of expert panelist 
indicator-level quality and relevance ratings is provided in Figure 8-1. 
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Figure 8-1. 


Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Northeast & Islands 
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Figure Reads: 6.0 percent of the Indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on 
a 5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 266 and 134 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 402 and 204 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 15 

N (Reports) = 17 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: Northwest 



Regional Educational Laboratory Northwest serves a region that includes the following states: 

• Alaska; 

• Idaho; 

• Montana; 

• Oregon; and 

• Washington. 

For the 2006-201 1 contract period, REL Northwest was headquartered at Education Northwest in 
Portland, Oregon. At the time of data collection, that organization, previously known as Northwest 
Regional Educational Laboratory, had served as a REL since 1966. 

Description of Projects and Activities 

In June 2010, a member of the evaluation study team interviewed the REL Northwest director and 
one staff member. REL staff members were asked to provide up to three examples of the work their 
REL had completed under each mission specified in ESRA. 48 Other than documenting REL reports 
published by IES, the evaluation study team made no attempt to verify independently the reported 
information on REL activities, so the accuracy and completeness of this information is unknown. 

Mission #1: Training or Technical Assistance 

REL Northwest staff highlighted the following projects and activities occurring under this mission: 

• Analysis of Student Enrollment and Teacher Endorsements for High School Science 
and Math (Fast Response Project /FRP). State education agencies (SEAs) requested that 
REL Northwest address the following question: As states follow recommendations to have 
more rigorous course work for all students, particularly in the areas of math and science, 
how will school districts and schools respond regarding the capacity of the teacher work 


48 For a more detailed description of the methodology, see chapter 2. 
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force to effectively instruct students? According to the REL, this issue was particularly 
relevant given the high proportion of rural and smaller districts in the REL Northwest 
region. The first work the REL completed in this area was for Oregon and, at the time of the 
interview, the REL was embarking on similar work for Washington. 

• Response to Intervention (Rtl) (FRP). The initial product in this group was a report. 

Models of Response to Intervention in the Northwest Region States, describing models of Rtl in the 
states in the REL’s region. The REL followed up the report with a three-part Bridge Event 
webinar series in spring 2010 to provide additional access to the report. REL Northwest did 
this work in coordination with the Northwest Regional Comprehensive Center 49 . 

• Bridge Events. Bridge Events on improving the measurement of achievement gaps and 
trends focused on a specific state in the REL region. The Bridge Events in this series 
brought researchers, educators, and practitioners together to talk about research findings 
regarding commonly used accountability measures and how these findings can support or 
influence policy and practice around issues of achievement gaps and trends, using state- 
specific technical assistance data in each presentation to engage the participants. The REL 
held the first Bridge Event in March 2009 for Washington, the second event in October 
2009 for Alaska, and the third event in August 2010 for Oregon. For the events, the REL 
partnered with other organizations, such as the Alaska Association of School 
Administrators 50 and the Confederation of Oregon School Administrators, 51 to maximize 
attendance. In addition to these Bridge Events, REL Northwest, at the time of the interview, 
had held Bridge Events on seven additional topics: 

1 . Encouraging Girls in Math and Science 

2. Turning Around Low-Performing Schools: A Research Forum for Northwest 
Region Leaders 

3. Understanding the Evidence; Impact Evaluation of the U.S. Department of 
Education’s Student Mentoring Program 

4. Dropout Prevention: A Research Forum for Northwest Region Leaders 

5. What Does Research Tell Us About Academic Enrichment in After School 
Settings? 

6. School Turnarounds: A Forum for Montana Educators 

7. Improving Adolescent Literacy: Effective Classroom and Intervention Practices 


49 http:/ / nwrcc.educationnorthwest.org/ 

50 http://www.alaskaacsa.org/AASA/? 

51 http:/ /www.cosa.k!2.or.us/ 
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Mission #2: Dissemination 

In addition to the three examples of dissemination provided below, REL Northwest indicated that 
in several instances, it elected to use corporate funds to do some additional dissemination work that 
furthered the REL mission. Examples of this include work with the organization’s corporate 
magazine and follow-up web work for the American Recovery and Reinvestment Act 52 (ARRA) 
webinar described in the third bullet below. REL Northwest staff highlighted the following projects 
and activities occurring under this mission: 

• Bridge Event Promotion and Documentation. REL Northwest described a multifaceted 
dissemination strategy that includes distributing printed fliers, press releases, email “blasts” 
(emails sent simultaneously to large groups), videos of events on its website, and articles in 
its professional magazine. Education Northwest Magazine 13 . For example, for a Bridge Event in 
Alaska on the IES practice guide describing how to turn around chronically low-performing 
schools, the REL mailed 499 hardcopy invitations to Alaska school principals and district 
superintendents, SEA personnel, and other policymakers. The REL also advertised the event 
on its website and in its magazine. REL Northwest reported that 68 school leaders attended 
the event. For those who were unable to attend, the REL provided video segments on its 
website. 

• Fast Response Project Dissemination. REL Northwest used press releases and other 
dissemination options, such as its website, email blasts, and magazine to publicize the release 
of FRPs. It then followed up this effort with presentations at conferences and fomms of 
stakeholders. The REL provided an example of this effort with its FRP, Indian Education 
Policies in Five Northwest Region States. It distributed press releases and the IES-published 
report to newspapers, organizations interested in Indian education policy and the 
mainstream education press (e.g.. Education Daily and Education Week). According to the REL, 
Education Daily 54 covered the release of this report and REL Northwest presented the FRP at 
the Alaska Native Education Association" 5 and the Montana Indian Education Association" 6 
annual conferences. At both of these events, REL Northwest included a copy of the 
executive summary from the FRP in the conference packet. 

• Webinar on Using ARRA Resources to Leverage Reform. REL Northwest collaborated 
with the Northwest Regional Comprehensive Center to respond to several information 
requests about ARRA from the five states in the region. For this 90-minute webinar, the 
REL had senior leaders from ED discuss ARRA implementation to help people in the 
region better understand the policy environment related to this legislation. REL Northwest 
described the webinar participants as primarily from the Northwest, but some were from 


52 http://www.recovery.gov/Pages/default.aspx 

53 http: / / educationnorthwest.org/ edunw-magazine 

54 http:/ /www.educationdaily.com/GE/ splash.jsp 

55 http: / / aneaonline.org/ 

56 http: / / www.mtiea.org/ 
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elsewhere in the country. To recruit participants, the REL sent an email to the approximately 
8,000 members on its general listserv and posted a number of early ARRA-related guidance 
documents from ED on its website. In addition, the REL noted that it used corporate, not 
REL, funds to create an online resource that provided materials regarding ARRA to the 
states in the region (e.g., federal guidance and links to ARRA fomms). 

Mission #3: Needs Assessment or Needs Sensing 

Starting in 2008, the REL staff members reported that they added state outreach coordinators 
(SOCs) to work with each of the five states in the region. The SOCs served as a point of contact for 
the SEA, educational organizations in the state, and schools requesting services from REL 
Northwest. According to the REL, the SOCs provided REL Northwest with ongoing feedback on 
the needs in their states and were involved in planning activities and projects. REL Northwest staff 
highlighted the following projects and activities occurring under this mission: 

• Regional Needs Assessment Survey. REL Northwest administered this Office of 
Management and Budget-cleared needs assessment survey of superintendents in the region 
in 2007 and 2009 of the contract and also included a random sample of principals and 
teachers in the survey. 

• State Forums. REL Northwest held state forums every other year that included state boards 
of education; the governor’s aide for education; the education subcommittee members of the 
legislature; local school district representatives; business groups; representatives from higher 
education; and representatives from various associations, including the state school 
administrators, a range of educators concerned with mral, bilingual, Indian, and teacher 
education, and school librarian associations. The REL staff members reported that the 
purpose of these forums was to gather additional data on the needs of the region, beyond 
the data obtained through the regional needs assessment survey. 

• Topical Forums. According to the REL, once it determined that a topic was relevant to the 
area, it held a topical forum, which was an invitational meeting on the special interests 
around that issue. REL Northwest held one to two topical forums in each year of the 
contract. As an illustration of this type of activity, REL Northwest described bringing 
together representatives of the land grant universities in the region with major 
responsibilities for service to the field and teacher education. The REL staff members stated 
that two things came out of this fomm. First, the staff members noted that these universities 
formed a consortium so they could continue to work together in the future. Second, the staff 
members reported that the universities identified Indian education as an important issue on 
which they needed to focus some of their resources. 

Mission #4: Carrying Out Applied Research Projects 

REL Northwest staff highlighted the following projects and activities occurring under this mission: 

• An Investigation of the Impact of a Traits-Based Writing Model on Student 
Achievement (Impact Study). The traits-based writing model is not an alternative writing 
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curriculum designed to replace existing writing programs in schools, but rather an additional 
set of tools to aide in conceptualizing, assessing, and describing the qualities of writing. This 
model targets the seven traits of effective writing: ideas: organization, voice, word choice, 
sentence fluency, conventions, and presentation. According to the REL, the traits-based 
writing model was in widespread use in the Northwest region, and although there were few 
studies demonstrating its effectiveness, its popularity was based on empirical arguments for 
the underlying model. This 5-year study used random assignment at the school level to study 
the performance of students in fifth grade in Oregon. REL Northwest reported that, on 
average, two fifth-grade classrooms participated in each school, yielding 74 classrooms in 
each of the treatment and control groups. 3 

• An Experimental Study of the Project CRISS Reading Program on Ninth-Grade 
Achievement in Rural Schools (Impact Study). This study examined the intervention, 
Creating Independence through Student-owned Strategies 58 (CRISS), a professional 
development intervention for high school teachers in the core subjects. The school-level 
random assignment study included approximately 8,000 ninth-grade students in 60 rural high 
schools — 30 treatment schools and 30 control schools. Teachers in the treatment schools 
received Project CRISS in the 2007-08 and 2008-09 school years, while the teachers in the 
control group did not receive the intervention until 2009-10, after all student and teacher 
data were collected. The ninth-grade students in all schools were tested in reading in fall 
2008 and spring 2009. 59 

Mission #5: Providing Applied Research Projects in a Usable Format 

REL Northwest staff highlighted the following projects and activities occurring under this mission: 

• Conducting Brie Engs. REL Northwest cited as an example of research in a usable form a 
session it co-presented with REL Southeast at the Council of the Great City Schools’ 6 " 
annual meeting of research, curriculum, and instruction directors of more than 60 member 
districts. In this presentation, REL Northwest provided information about the REL system 
and described the free services available. 

• Repackaging Fast Response Projects. Using the original FRPs, REL Northwest created 
shorter products that were each written for a specific target audience. According to the REL, 
the goal was to reach users who did not find the full report accessible by providing them 
with more user-friendly information. For example, with its FRP, New and Experienced Teachers 
in a School Reform Initiative , the REL stated that it had an announcement in the journal Reading 
Today, 6 ' as well as a feature story in the Education Northwest Magazine. As described earlier 
under dissemination, REL Northwest also distributed information about its available 
products through a variety of means, including email blasts and press releases. 


57 IES published a report from this study in December 201 1 

58 http:/ /www.projectcriss.com/l-what-we-are 

59 IES published a report from this study in April 2011. 

60 http:/ / www.cgcs.org/ site/default.aspx?PageID=l 

61 http:/ / www.reading.org/ General/Publications/blog/About_Reading_Today.aspx 
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• Reference Desk. Via Ask-A-REL, constituents could submit questions to REL Northwest 
through the REL’s website. Once these requests were received, they were entered into the 
Reference Desk, a database that houses questions received by the REL with available 
responses to questions. If the Reference Desk did not contain a response to the requester’s 
question, the REL staff members could provide a response through several means, including 
a literature review or other reference information. For example, the REL staff members 
described work they conducted for several small school districts on 4-day school weeks. 

Staff members at the Reference Desk prepared a literature review of the best available 
information concerning the pros and cons of this alternative school schedule. The REL then 
expanded the literature review for an article in the Education Northwest Magazine using 
corporate, not REL, funds. 

Mission #6: Coordination or Collaboration 62 

REL Northwest staff highlighted the following projects and activities occurring under this mission: 

• Achievement Gap Patterns of Grade 8 American Indian and Alaska Native Students 
in Reading and Mathematics. The REL staff members stated that REL Northwest was 
the leading partner in creating this collaboration with seven other RELs. The dissemination 
strategy included collaborative briefings on recent National Assessment of Educational 
Progress results for American Indian and Alaska Native students, as well as a webinar with 
the Comprehensive Centers responsible for the western states, and a presentation at the U.S. 
Office of Indian Education Partnership Conference. 

• Rural Working Group. Since REL Northwest serves a sizable rural constituency, the REL 
staff members reported that REL Northwest participated in the REL-wide mral working 
group by attending the monthly conference calls. The REL was also involved in planning 
other joint events across the RELs, including a conference that was planned for November 
2010 . 

• Data Working Group. REL Northwest participated in this REL-wide working group and 
volunteered to be the lead REL in submitting a proposal to IES for an FRP on using 
longitudinal data to inform an examination of effective teachers. 

Mission #7: School Finance Systems 

REL Northwest interpreted this mission to include projects directed at helping schools and districts 
deal with economic crisis. REL Northwest staff highlighted the following projects and activities 
occurring under this mission: 

• What the Research Says (and Doesn ’t Say): the Four-Day School Week (Literature 
Review). Described under Mission #5, in response to several requests from small school 
districts exploring the option of switching to a 4-day school week as a potential way to 
reduce costs, the Reference Desk prepared a literature review of the available information. 


62 Under other missions the REL described several activities it conducted in coordination with the Northwest Comprehensive Center, 
which is housed in the same company and the Alaska Comprehensive Center operated by SERCC. 
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• What the Research Says (and Doesn’t Say): Consolidation of Schools (Literature 
Review). According to the REL staff members, school consolidation was an important issue 
to states with numerous small, often mral school districts. The request for this literature 
review came from the REL Northwest region through the Reference Desk, which compiled 
existing research to address the following questions: Does consolidation reduce costs? What 
is the impact on students? 

• Responding to the Economic Crisis: Considerations for School Administrators 
(Literature Review). Several people in REL Northwest’s region contacted the Reference 
Desk with the question, “What does the literature say about what you do when there is an 
economic crisis?” To answer this question, REL Northwest reviewed recommendations 
from the business community in this area and created a bibliography of citations. In 
addition, the REL produced an issue of Northwest Education Magazine under the title 
“Weathering the Storm,” which provided additional information on the subject. 

Mission #8: Alternative Administrative Structures 


REL Northwest staff highlighted the following project or activity occurring under this mission: 


Bridge Event on Charter Schools. This Bridge Event took place in spring 2010 in Idaho 
and, according to REL staff members, illustrated the REL’s efforts to take Bridge Events 
beyond the IES practice guides. A representative from the National Center on School 
Choice 63 presented two publications from that organization — one on charter school 
outcomes and one on research related to school choice. The presentation highlighted the 
leading studies on school choice and described how to evaluate charter school research. This 
event was held at the request of charter school leaders and other stakeholders in the state. 
REL Northwest reported that approximately 80 people attended the event. 


Mission #9: School Improvement Plans 


REL Northwest staff highlighted the following projects and activities occurring under this mission: 

• Bridge Events on Turning Around Schools. The REL staff members noted that with 
more than 1,000 Title I 64 schools in the region labeled as in need of improvement, school 
turnaround is a large and growing issue for the states in the region. In response to this need, 
REL Northwest conducted three turnaround Bridge Events with the Comprehensive Center 
on Innovation and Improvement. 65 The REL reported that approximately 220 people 
attended these events. Following the events, all of the tools and materials distributed during 
the events were posted on the REL Northwest website. 

• How Northwest Region States Are Responding to Schools in Need of Improvement 
(FRP). REL Northwest created a report for regional policymakers that described the state 


63 http:/ / www.vanderbilt.edu/ schoolchoice/ 

64 http:/ / www2.ed.gov/ policy/ elsec/leg/esea02/pgl.html 

65 http: / / www.centerii.org/ 


Evaluation of the Regional Educational Laboratories: Interim Report 

89 


systems of support to help schools in need of improvement in place in each of the states in 
the region. 

• Topical Forum on School Support Teams. REL Northwest convened representatives 
from all five Northwest SEAs and their school support team members to share information 
about the REL’s roles and functions in school support, as well as to identify needs in the 
region related to school and district improvement. REL Northwest held this fomm in 
cooperation with the Northwest Regional and the Alaska 66 Comprehensive Centers in July 
2008. 

Mission #10: Technology in Education 

REL Northwest staff highlighted the following project or activity occurring under this mission: 

• Geographic Information System Mapping Project. REL Northwest partnered with the 
Montana Parent Information Resource Center 6 ' to examine the relationship between the 
concentration of children and intensity of child-related services in the state. The product of 
the partnership was a county-by-county map of Montana that overlaid the population 
concentration with related service availability. 

Quality and Relevance of Fast Response Projects 

As of December 1, 2009, 38 percent (15) of REL Northwest’s 40 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 9-1). Forty-eight percent (10) of REL Northwest’s 21 submissions for FRP 
reports were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 


By December 1, 2009, REL Northwest had 10 FRP reports accepted and published by IES: 

Table 9-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Northwest and all 10 RELs combined as of December 1, 2009 



Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 

Northwest 

40 

15 

38 

All RELs 

297 

137 

46 

Reports 

Northwest 

21 

10 

48 

All RELs 

166 

92 

55 


Table Reads : Of the 40 REL Northwest proposals submitted, 15 (38 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


66 http: / / www.alaskacc.org/ about 

67 http:/ / montanapirc.com/ 
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1 . Analysis of the Title IIB Mathematics and Science Partnerships in the Northwest Region (June 2007); 

2. Coach Can Mean Many Things: Fine Categories ofUteray Coaches in Reading First (June 2007); 

3. Supplemental Educational Services and Implementation Challenges in the Northwest Region States (June 
2007); 

4. Flow Northwest Region States Are Responding To Schools In Need of Improvement (August 2007) ; 

5. Indian Education Policies in Five Northwest Region States (October 2007); 

6. Parent Involvement Activities in School Improvement Plans in the Northwest Region (October 2008); 

7. Achievement Gap Patterns of Grade 8 American Indian and Alaska Native Students in Reading and 
Mathematics (July 2009); 

8. Analysis of the Developmental Functioning of Early Intervention and Early Childhood Special Education 
Populations in Oregon (August 2009); 

9. Models of Response to Intervention in the Northwest Region States (September 2009); and 

10. New and Experienced Teachers in a School Reform Initiative: The Example of Reading First (November 
2009). 

Table 9-2 shows the expert panel quality and relevance ratings for these IES-published reports from 
REL Northwest and for the ten corresponding proposals. <>8 Average quality and relevance ratings 
for proposals and reports from all 10 RELs are provided for reference. Quality rating categories 
ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged from 1 
(“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and IES-published 
reports received a mean quality rating of 3.81 on a 5-point scale. FRP proposals from REL 
Northwest received a mean quality rating of 3.47, and the corresponding IES-published reports 
received a mean quality rating of 3.72. Among the 10 REL Northwest proposals, mean quality 
ratings ranged from 2.82 to 4.39. Among the 10 REL Northwest IES-published reports, mean 
quality ratings ranged from 2.96 to 4.54. 

With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 


68 Appendix A describes in detail the indicators that make up the quality and relevance dimensions. 
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Table 9-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Northwest and all 10 RELs combined 
(on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All REL FRPs ( N = 75) 

3.24 

3.39 

All Northwest FRPs (N = 10) 

3.47 

3.47 

Achievement Gap Patterns of Grade 8 American Indian 
and Alaska Native Students in Reading and Mathematics 

3.78 

3.78 

Analysis of the Developmental Functioning of Early 
Intervention and Early Childhood Special Education 
Populations in Oregon 

4.39 

4.11 

Analysis of the Title MB Mathematics and Science 
Partnerships in the Northwest Region 

3.50 

3.56 

Coach Can Mean Many Things: Five Categories of Literacy 
Coaches in Reading First 

3.06 

3.78 

How Northwest Region States Are Responding To Schools 
In Need of Improvement 

3.56 

2.89 

Indian Education Policies in Five Northwest Region States 

2.82 

2.78 

Models of Response to Intervention in the Northwest 
Region States 

3.44 

3.44 

New and Experienced Teachers in a School Reform 
Initiative: The Example of Reading First 

4.00 

4.00 

Parent Involvement Activities in School Improvement 
Plans in the Northwest Region 

3.06 

3.44 

Supplemental Educational Services and Implementation 
Challenges in the Northwest Region States 

3.12 

2.89 

Reports 


All REL FRPs (N = 91) 

3.81 

3.64 

All Northwest FRPs ( N = 10) 

3.72 

3.71 

Achievement Gap Patterns of Grade 8 American Indian 
and Alaska Native Students in Reading and Mathematics 

3.54 

3.17 

Analysis of the Developmental Functioning of Early 
Intervention and Early Childhood Special Education 
Populations in Oregon 

4.33 

4.42 

Analysis of the Title MB Mathematics and Science 
Partnerships in the Northwest Region 

3.67 

3.17 

Coach Can Mean Many Things: Five Categories of Literacy 
Coaches in Reading First 

3.54 

4.17 

How Northwest Region States Are Responding To Schools 
In Need of Improvement 

3.52 

4.08 

Indian Education Policies in Five Northwest Region States 

3.30 

3.17 
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Table 9-2. 


Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Northwest and all 10 RELs combined 
(on a 5-point scale with 5 being the highest) (continued) 


Mean Ratings 

Quality 

Relevance 

Models of Response to Intervention in the Northwest 
Region States 

3.67 

3.33 

New and Experienced Teachers in a School Reform 
Initiative: The Example of Reading First 

4.08 

3.58 

Parent Involvement Activities in School Improvement 
Plans in the Northwest Region 

4.54 

4.67 

Supplemental Educational Services and Implementation 
Challenges in the Northwest Region States 

2.96 

3.33 


Table Reads : For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-publlshed FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects 
(Supplemental Educational Services and Implementation Challenges in the Northwest Region States For lES-published reports); Rubric 
and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For Proposals) (Appendix 
A). 


mean rating of 3.64 for the actual reports. REL Northwest proposals received a 
mean relevance rating of 3.47, and the corresponding IES-published reports received a mean 
relevance rating of 3.71. Among the 10 REL Northwest proposals, the mean relevance ratings 
ranged from 2.78 to 4.11. Among the 10 REL Northwest IES-published reports, the mean relevance 
ratings ranged from 3.17 to 4.67. Additional detail on the distribution of expert panelist 
indicator-level quality and relevance ratings is provided in Figure 9-1. 
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Figure 9-1. 


Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Northwest 


s» 
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Proposals 

Reports 


Proposals 

Reports 



Ratings 

■ 1 (Very weak) 

■ 2 (Weak) 

■ 3 (Adequate) 

4 (Strong) 

5 (Very strong) 

■ 1 (Not relevant) 

■ 2 (Marginally relevant) 

■ 3 (Adequate) 

4 (Relevant) 

5 (Highly relevant) 


Figure Reads: 14.3 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on 
a 5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 175 and 90 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 236 and 120 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 10 

N (Reports) = 10 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: Pacific 



Regional Educational Laboratory Pacific serves a region that includes the following geographic 
locations: 

• American Samoa; 

• Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap); 

• Guam; 

• Hawaii; 

• Northern Mariana Islands; 

• Republic of the Marshall Islands; and 

• Republic of Palau. 

For the 2006-201 1 contract period, REL Pacific was housed at Pacific Resources for Education and 
Learning (PREL) in Honolulu, Hawaii. PREL had held previous REL contracts. 

Description of Projects and Activities 

In May 2010, a member of the evaluation study team interviewed the REL Pacific director. The REL 
director was asked to provide up to three examples of the work the REL had completed under each 
mission specified in ESRA. 69 Other than documenting REL reports published by IES, the evaluation 
study team made no attempt to verify independently the reported information on REL activities, so 
the accuracy and completeness of this information is unknown. 

Mission #1: Training or Technical Assistance 

REL Pacific staff highlighted the following projects and activities occurring under this mission: 

• Bridge Events. REL Pacific sought to reach a large constituent base by using web-based 
software, “Elluminate,” for online Bridge Events. Since REL Pacific’s constituents are in 


69 For a more detailed description of the methodology, see chapter 2. 
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various time zones, these events were typically repeated several times throughout the day. 
These sessions were also recorded and made available to clients for formal follow-up 
sessions and self-directed professional development for clients who were unable to 
participate in “real time.” At the time of the interview, REL Pacific had held Bridge Events 
on eight topics: 

1 . Turning Around Chronically Low-Performing Schools 

2. Improving Adolescent Literacy: Effective Classroom and Intervention Practices 

3. Dropout Prevention 

4. Assisting Students Struggling With Reading: Response to Intervention (Rtl) and 
Multi-Tier Intervention in the Primary Grades 

5. Assisting Students Struggling With Reading: Response to Intervention (Rtl) for 
Elementary and Middle Schools 

6. Structuring Out-of-School Time to Improve Academic Achievement 

7. Using Student Achievement Data to Support Instmctional Decision-Making 

8. Effective Literacy and English Language Instruction for English Learners in the 
Elementary Grades 

• AskPREL. Similar to Ask-A-REL, “AskPREL” provided constituents with an online tool 
on PREL’s website to ask REL Pacific questions. According to the director, the REL 
received inquiries from various constituents, such as SEAs, principals, and teachers. Once 
the REL received a question, the appropriate staff member sent a response or the 
appropriate materials. If the REL was unable to answer a question, the REL contacted other 
RELs or sources that could answer the inquiry. While AskPREL was primarily a means of 
dissemination, the R EL director also described it as a technical assistance tool, since the REL 
used it to provide answers to questions as well as connect clients to experts in the pertinent 
field. 

Mission #2: Dissemination 

The director reported that while technology is an important conduit for information dissemination, 
some locations in the Pacific region lacked the necessary technology to receive the REL’s 
information electronically. As a result, REL Pacific also printed hardcopy versions of its materials 
that were then mailed to recipients. REL Pacific staff highlighted the following projects and 
activities occurring under this mission: 

• PREL’s Website. The website provided the REL’s audience with access to the REL’s work, 
including its reports and a link to the national REL webpage. 
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• PaciEc Educator. This is a newsletter that the REL sent out biannually to all teachers in the 
region (approximately 15,000-19,000 teachers). REL Pacific also made the Pacific Educator 
available through its website. In 2010, the REL planned to publish three issues of the Pacific 
Educator. One publication was to be a special edition dedicated to the REL Pacific work and 
products. The REL director noted that the status of technology infrastructure development 
and access to Internet in the region meant that print dissemination of the Pacific Educator was 
many educators’ only source of up-to-date information related to both research and best 
practices. 

• AskPREL. Described under Mission #1, AskPREL provided constituents with an online 
tool on PREL’s website to ask REL Pacific questions. After the REL received a question, a 
staff member sent a response or the appropriate materials. 

Mission #3: Needs Assessment or Needs Sensing 

REL Pacific conducted needs sensing in multiple ways, such as drawing on advice from the advisory 
groups described below. REL Pacific also drew on the knowledge of the REL Technical Working 
Group (TWG) and the REL Reading Advisory Panel to gain an understanding of its constituents’ 
needs. According to the REL director, the TWG was particularly helpful in discussing ways that the 
REL’s impact studies could meet regional needs. In addition, the REL director noted that the 
AskPREL database was used for needs sensing. U sing that database, REL Pacific ran a monthly 
report that highlighted what issues had been requested most frequently by the REL’s constituency. 
REL Pacific also conducted needs sensing by drawing on data that it gathered via PREL’s Pacific 
Data Center. More generally, the REL conducted informal needs assessments when talking with 
constituents. REL Pacific staff highlighted the following projects and activities occurring under this 
mission: 

• Governing Board. The director stated that the REL’s Governing Board met three to four 
times a year. At these meetings, the board members discussed the most important needs in 
the region. 

• PaciEc Curriculum and Instruction Council (PCIC). The PCIC comprised all heads of 
curriculum in the region, and met twice a year. At these meetings, the chiefs discussed the 
issues that were important to the region and ways that REL Pacific could help its 
constituency. This group also assisted PREL in aligning the services of the REL, the Pacific 
Regional Comprehensive Center (PRCC), and other programs to leverage resources and 
most effectively address the full spectmm of the region’s needs. 

• Researcher/Evaluator Meeting. The REL met twice a year with a group of researchers 
and evaluators comprising one expert from each entity in the region. The director stated that 
this meeting helped the REL to gather information on ways to best meet the needs of all of 
the region’s constituents. The director also noted that the meeting helped the REL to build 
capacity to do research and evaluations in the region’s various localities, because some of the 
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locations lacked basic education data, for example, the number of students in each grade in 
their system. 70 

Mission #4: Carrying Out Applied Research Projects 

REL Pacific staff highlighted the following project or activity occurring under this mission: 

• Pa ci£c CHILD (Impa ct Study). REL Pacific conducted a randomized controlled trial that 
focused on principles-based professional development for literacy (reading comprehension) 
in grades 4 and 5. This study built on research that the REL completed under the previous 
REL contract on professional development for early literacy in grades K-3 and also explored 
literacy issues as they relate to English language learners (ELLs) in the region. According to 
the director, grades 4 and 5 are a particularly important transitional period for many students 
in REL Pacific’s region because most students are taught in their native language through 
third grade, but transition to classroom instruction in English in fourth grade. 

Mission #5: Providing Applied Research Projects in a Usable Format 

While REL Pacific had published six Fast Response Projects at the time of the interview, the REL 
often found that the data it needed to support the development and recommendations in potential 
FRPs was not available. To better support its ability to obtain data in the region, REL Pacific 
focused many of its efforts on developing a “culture of data,” meaning that the REL worked to help 
its constituents understand the value and importance of data in the Pacific region. REL Pacific staff 
highlighted the following projects and activities occurring under this mission: 


• Fast Response Projects. REL Pacific FRPs encompassed multiple themes, including 
student performance and teacher quality. REL Pacific had completed and distributed six 
FRPs to its constituency at the time of the interview and had plans to publish three 
additional FRPs. 

• Impact Study Brief. At the time of the interview, the REL Pacific director stated that the 
REL would publish a short report summarizing the results of its impact study discussed 
under Mission #4, Pacific CHILD. 

• AskPREL. As was described in detail under Mission #1, REL Pacific generated quick 
responses for constituents who asked questions to PREL via AskPREL on the PREL 
website. 

Mission #6: Coordination and Collaboration 

Since PREL held other federal education contracts, the REL often worked collaboratively with 
related organizations, such as the PRCC, the Territories & Freely Associated States Education 


70 Note that some entities are not required to submit data to the Common Core of Data. 
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Grant, the equity center, and two Parent Information Resource Centers. The REL director said that, 
unlike other RELs, REL Pacific and the PRCC served identical regions of the United States (the 
only REL and Comprehensive Center pair that had matching regions during the 2006-201 1 contract 
period), and since they were both embedded in PREL, they were well positioned for collaboration, 
including a single combined governing board. The director reported that, while the various grants 
within PREL enabled the leveraging of resources, PREL also recognized the clearly defined mission 
of the REL contract. Program directors within PREL met at least once a month to discuss their 
contract activities and ways to leverage resources. For instance, the REL often worked with the 
PRCC on Bridge Events. 


REL Pacific also collaborated with other RELs. In addition to the activities described below, REL 
Pacific worked with the other RELs to produce an FRP on the achievement gap patterns of eighth- 
grade American Indian and Alaska Native students. REL Pacific’s director reported that REL 
Northwest was the lead partner on the report, but REL Pacific actively participated. REL Pacific 
staff highlighted the following projects and activities occurring under this mission: 


• Bridge Events. REL Pacific worked with the PRCC to conduct several Bridge Events, 
including one on Response to Intervention. For this event, the Comprehensive Center 
helped recruit state-level personnel to attend, while REL Pacific was able to focus on the 
practitioners and the event message (including the practice guide, author, and research-based 
recommendations) . 

• Data, Data Collection, and Data Management. At PREL, staff members from many of 
the organization’s U.S. Department of Education (ED) contracts worked together around 
issues of data, data collection, and data management to improve data quality in the Pacific 
region, with a common goal of building a culture around data in the region. 

• Collaboration and Coordination. REL Pacific participated in the National Laboratory 
Network (NLN) website, Thursdays @ 3, the various working groups (i.e., data, 
dissemination, mral, etc.), and the REL director meetings. According to the director, REL 
Pacific also led several Thursday @ 3 meetings. 

Mission #7: School Finance Systems 

REL Pacific staff highlighted the following project or activity occurring under this mission: 

• Data Collection, Including Data on School Finance. The REL director reported that, in 
the future, REL Pacific wanted to collect data in the Pacific region to enable the REL to 
answer the following questions regarding the cost of education: (1) What are the region’s 
expenditures on education? (2) What proportion of the region’s economy is spent on 
education? (3) What types of educational programs are states, territories, and schools 
funding? 
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Mission #8: Alternative Administrative Structures 

According to the REL director, all state education agencies (SEAs) of the states and territories in 
REL Pacific’s region also act as local education agencies (LEAs), and most entities in the region 
have a top-down structure where few have school boards or unions. REL Pacific did not do much 
work in the area of alternative administrative structures, as this topic was not considered as relevant 
to the Pacific region as to other regions. 

Mission #9: School Improvement Plans 

REL Pacific engaged in various projects and activities related to school improvement plans. The 
director reported that PREL was one of Hawaii’s school improvement service providers, so this 
mission was embedded throughout the organization. The director also noted that REL Pacific had a 
regional presence in conducting professional development, including principles-based professional 
development that focused on school improvement plans. REL Pacific staff highlighted the following 
projects and activities occurring under this mission: 

• Bridge Event on Turning Around Low-Performing Schools. REL Pacific presented a 
Bridge Event based on a What Works Clearinghouse practice guide that focused on how to 
improve low-performing schools. 

• Bridge Event Follow-Up. Following the Bridge Event on dropout prevention, REL Pacific 
provided technical assistance to one of its constituents on ways to use data to better 
understand and improve dropout rates. 

• Pacific CHILD (Impact Study). As was described in more detail under Mission #4, REL 
Pacific conducted a study that focused on principles-based professional development for 
literacy (reading comprehension) in grades 4 and 5. This study also explored literacy issues as 
they relate to ELLs and was pertinent to school improvement. 

Mission #10: Technology in Education 

According to the REL director, it was difficult for REL Pacific to conduct projects on technology in 
education since technology costs in the Pacific region were extremely high, and the region’s schools 
lacked technology. Given these technological limitations, REL Pacific needed to produce flexible 
products and activities (e.g., distribute paper copies of reports rather than rely solely on Internet 
distribution). However, the REL director commented that this area was one in which REL Pacific 
would be able to play a regional role in the future by facilitating the growth of technology in 
education. REL Pacific staff highlighted the following projects and activities occurring under this 
mission: 
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• Technical Assistance Requests. At the time of the interview, the REL director said that 
the REL, in conjunction with the Comprehensive Center, recently received a request from 
the Northern Marianas to provide it with a network of people implementing distance 
learning effectively. 

• Webinars. Given the geographic distribution of its constituents throughout the Pacific 
region, REL Pacific frequently used the webinar tool, “Elluminate,” to conduct Bridge 
Events and other webinars. 

• Paper on Cyber Bullying. REL Pacific prepared a paper on cyber-bullying that did not go 
through the standard IES review process. REL Pacific had the paper available for download 
on the PREL website with a disclaimer. 

Quality and Relevance of Fast Response Projects 

As of December 1, 2009, 41 percent (7) of REL Pacific’s 17 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 10-1). Forty-two percent (5) of REL Pacific’s 12 submissions for FRP 
reports were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 


By December 1, 2009, REL Pacific had five FRP reports accepted and published by IES: 

1 . English Language Proficiency Assessment in the Pacific Region (June 2007); 

2. Preparing and Licensing High Quality Teachers in Pacific Region jurisdictions (September 2007); 

3. A Status Report on Middle School Mathematics Assessment and Student Achievement in the Pacific Region 
(January 2008); 

4. The Status ofi Large-Scale Assessment in the Pacific Region (July 2008); and 

5. The Status of the Preparation and Hiring of School Principals in the U .S '.-Affiliated Pacific Region 
(August 2008). 


Evaluation of the Regional Educational Laboratories: Interim Report 

101 


Table 10-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Pacific and all 10 RELs combined as of December 1, 2009 



Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 

Pacific 

17 

7 

41 

All RELs 

297 

137 

46 

Reports 

Pacific 

12 

5 

42 

All RELs 

166 

92 

55 


Table Reads : Of the 17 REL Pacific proposals submitted, 7 (41 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


Table 10-2 shows the expert panel quality and relevance ratings for these IES-published reports 
from REL Pacific and for the five corresponding proposals. Average quality and relevance ratings 
for proposals and reports from all 10 RELs are provided for reference. Quality rating categories 
ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged from 1 
(“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and IES-published 
reports received a mean quality rating of 3.81 on a 5-point scale. FRP proposals from REL Pacific 
received a mean quality rating of 2.54, and the corresponding IES-published reports received a mean 
quality rating of 3.14. Among the five REL Pacific proposals, mean quality ratings ranged from 2.22 
to 2.78. Among the five REL Pacific IES-published reports, mean quality ratings ranged from 2.88 
to 3.38. 

With respect to relevance, all 10 RELs combined received mean ratings of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 
mean rating of 3.64 for the actual reports. REL Pacific proposals received a mean relevance rating of 
2.82, and the corresponding IES-published reports received a mean relevance rating of 3.30. Among 
the five REL Pacific proposals, mean relevance ratings ranged from 2.44 to 3.33. Among the five 


71 Appendix A describes in detail the indicators that make up the quality and relevance dimensions. 
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Table 10-2. Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Pacific and all 10 RELs combined 
(on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All REL FRPs (N = 75) 

3.24 

3.39 

All Pacific FRPs (N = 5) 

2.54 

2.82 

A Status Report on Middle School Mathematics 
Assessment and Student Achievement in the Pacific 
Region 

2.62 

2.56 

English Language Proficiency Assessment in the Pacific 
Region 

2.22 

2.44 

Preparing and Licensing High Quality Teachers in Pacific 
Region Jurisdictions 

2.53 

3.33 

The Status of Large-Scale Assessment in the Pacific 
Region 

2.78 

2.67 

The Status of the Preparation and Hiring of School 
Principals in the U.S.-Affiliated Pacific Region 

2.56 

3.11 

Reports 


All REL FRPs (N = 91) 

3.81 

3.64 

All Pacific FRPs (N = 5) 

3.14 

3.30 

A Status Report on Middle School Mathematics 
Assessment and Student Achievement in the Pacific 
Region 

3.17 

3.42 

English Language Proficiency Assessment in the Pacific 
Region 

2.88 

2.75 

Preparing and Licensing High Quality Teachers in Pacific 
Region Jurisdictions 

3.38 

3.83 

The Status of Large-Scale Assessment in the Pacific 
Region 

3.08 

2.92 

The Status of the Preparation and Hiring of School 
Principals in the U.S.-Affiliated Pacific Region 

3.17 

3.58 


Table Reads : For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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Figure 10-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Pacific 
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Figure Reads: 2.3 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 1 on a 5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 87 and 45 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 118 and 60 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 5 

N (Reports) = 5 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 


REL Pacific IES-published reports, the mean relevance ratings ranged from 2.75 to 3.83. Additional 
detail on the distribution of expert panelist indicator-level quality and relevance ratings is provided in 
Figure 10-1. 
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REL: Southeast 


11 


Regional Educational Laboratory Southeast serves a region that includes the following states: 

• Alabama; 

• Florida; 

• Georgia; 

• Mississippi; 

• North Carolina; and 

• South Carolina. 

For the 2006-201 1 contract period, REL Southeast was a university-based research center housed in 
the SERVE Center at the University of North Carolina - Greensboro. The SERVE Center had held 
the REL Southeast contract for the previous four cycles of funding. 

Description of Projects and Activities 

In June 2010, a member of the evaluation study team interviewed the REL Southeast director. The 
director was asked to provide up to three examples of the work the REL had completed under each 
mission specified in ESRA. 7 ~ Other than documenting REL reports published by IES, the evaluation 
study team made no attempt to verify independently the reported information on REL activities, so 
the accuracy and completeness of this information is unknown. 

Mission #1: Training or Technical Assistance 

REL Southeast staff highlighted the following projects and activities occurring under this mission: 

• Evidence-Based Education (EBE) Request Desk and Database. The REL director 
described the EBE Request Desk as a tool used by the REL to respond quickly to its 
constituents’ questions. REL Southeast also forwarded its EBE Request Desk responses to 
the National REL Reference Desk. The EBE Request Desk database housed constituents’ 


72 For a more detailed description of the methodology, see chapter 2. 
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questions and the lab’s responses to the questions. REL Southeast reported that, as of June 
2010, it had responded to 695 requests from state education agencies (SEAs), local education 
agencies (LEAs), individual schools, other labs, education organizations, and institutions of 
higher learning. The REL reported that it received approximately 10 to 12 requests a month. 
According to the director, request topics included engineering academies and English 
language learner (ELL) populations. In response to the requests, the REL produced formal 
“mini-reports,” written email responses, trainings, and/ or Bridge Events. 

• Bridge Events. REL Southeast conducted approximately one Bridge Event a month. For 
example, REL Southeast conducted a Bridge Event for the Georgia Department of 
Education regarding special education based on a REL Southeast Fast Response Project 
(FRP). The department recommended that all superintendents within Georgia participate in 
the event. After the superintendents participated, they asked that the REL also conduct the 
event for teachers within their districts. The REL director stated that to reach as large an 
audience as possible, the event was conducted via a webinar. At the time of the interview, 
REL Southeast had held Bridge Events on 12 topics: 

1. Examining Research to Frame Alabama’s Dropout Prevention 

2. State Policies and Procedures and Selected Local Implementation Practices in 
Response to Intervention (Rtf) in the Six Southeast Region States 

3. Using Research to Select Interventions in the Response to Intervention Process- 
Mathematics 

4. Determining the Effectiveness of Professional Development 

5. Methodology and Use of Early Warning Systems 

6. Assisting Students Struggling With Reading: Response to Intervention (Rtl) for 
Elementary and Middle Schools Practice Guide 

7. Assisting Students Struggling With Mathematics: Response to Intervention (Rtl) for 
Elementary and Middle Schools 

8. Assisting Students Struggling With Reading: Response to Intervention (Rtf) and 
Multi-Tier Intervention for Reading in the Primary Grades 

9. Improving Adolescent Literacy: Effective Classroom and Intervention Practices 

10. Using Data to Design and Evaluate Effective Professional Development 

11. Thinking Strategically Across ARRA Reform Initiatives (collaborated with other 
RELs) 

12. Using Data to Design and Evaluate Effective Professional Development 

• Evidence Based Education (EBE) Work. The REL director stated that the EBE work 
helped practitioners better interpret research and apply its findings. An example of REL 
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Southeast’s EBE work was the science academy network in Florida. The REL worked with 
multiple Florida districts on ways to use research to improve science education. To identify 
participants, each SEA in the region provided the REL with a list of school districts that the 
SEAs believed would be interested and in need of assistance from the REL in specific topic 
areas (e.g., high school dropouts) and then the REL contacted the superintendents of the 
districts to see if they were interested in participating. 

Mission #2: Dissemination 

REL Southeast staff highlighted the following projects and activities occurring under this mission: 

• SER VE Website. The website was used to distribute REL Southeast’s work to constituents, 
providing them with links of published reports and other resources. Constituents were also 
able to contact REL Southeast via the website. In addition, the website provided a link to the 
IES REL website. 

• Power Notes. These electronic newsletters were sent out monthly to the REL’s 
approximately 10,000 constituents (i.e., SEAs, LEAs, principals). The notes informed people 
of SERVE’s activities, including REL work. Power Notes (a.k.a. Powering Notes) were also 
available on SERVE’s website. Individuals could sign up via the website to have the Power 
Notes emailed to them monthly. 

Mission #3: Needs Assessment or Needs Sensing 

REL Southeast staff highlighted the following projects and activities occurring under this mission: 

• Market Research Firm-Conducted Needs Assessment. For four consecutive years 
under the 2006-201 1 contract, the REL hired a market research firm to gather needs 
assessment data for the REL by targeting large education events (i.e., conferences and 
conventions) held in each state in the southeast region. At these events, the firm held focus 
groups and asked constituents to talk about areas they believed were in greatest need of 
resources and/or help. In addition, the firm questioned the constituents to see how aware 
they were of REL Southeast’s resources. 

• Conversations with the SEA Chiefs. The REL director met annually with the SEA chiefs 
from each state in the REL Southeast region. When speaking with the chiefs and their staff 
members, the director used a needs sensing protocol to gauge what areas were in greatest 
need of REL Southeast services. 

• EBE Request Database. Each year, REL Southeast completed an analysis of constituents’ 
questions logged in its EBE Request Database. The REL categorized the questions by 
subject, and then conducted an analysis to determine what subjects were in highest demand 
of REL Southeast work and/ or assistance. 
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Mission #4: Carrying Out Applied Research Projects 


REL Southeast staff highlighted the following projects and activities occurring under this mission: 

• Alabama Math Science Technology Initiative (AMSTI) (Impact Study). This study 
evaluated the effectiveness of the AMSTI program in schools throughout Alabama. This 
program was launched as a statewide initiative focused on improving K-12 mathematics and 
science instruction, and schools applied to the state to participate. To test the impact of the 
initiative, schools applying to become part of AMSTI were randomly placed into the 
intervention group or into a control group. The achievement gains in the intervention and 
control groups were compared in order to determine the effectiveness of the program. 

• K-PAVEd for Success (Impact Study). K-PAVEd for Success is a model focused on 
improving early literacy, specifically vocabulary development, aimed at students from 
economically disadvantaged households. This randomized controlled trial evaluated the 
impact of the K-PAVEd model on teacher instruction and student vocabulary development. 
REL Southeast reported that kindergarten classes in 68 schools in the Mississippi Delta and 
surrounding areas participated in the study. 

Mission #5: Providing Applied Research Projects in a Usable Format 

REL Southeast staff highlighted the following projects and activities occurring under this mission: 

• Issues and Answers Reports. According to the REL director, REL Southeast produced 
many Issues and Answers reports for its constituents on educational topics relevant to the 
southeast region. Some examples of topics covered by REL Southeast’s reports include 
methods and instruments used in state and local school readiness evaluations, reducing 
stereotype threat in African American male students, Rtl, preparing teachers to work with 
students with disabilities, state policies and procedures, ELL students, and annual 
measurable achievement objectives. 

• Fast Response Projects. As of June 2010, IES had published six reports from FRPs 
conducted by REL Southeast. 

• Quick-Turnaround Data Analysis Projects. REL Southeast generated quick reports for 
constituents based on the analysis of SEA data. For example, Florida had questions 
concerning the mobility of its ELL students, and South Carolina requested a quick analysis 
of student achievement in its single gender schools. At the time of the interview, REL 
Southeast had plans to examine Florida’s state data and respond to it with a quick 
turnaround analysis (i.e., in 1 to 2 months). 

Mission #6: Coordination and Collaboration 

In addition to the activities listed below, the REL Southeast director indicated that REL staff 
members participated in Ask-A-REL, Thursdays @ 3, and other Task 6-led initiatives, such as the 
various REL working groups. The REL worked collaboratively with REL Mid-Atlantic and REL 
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Appalachia, as well as the regional Comprehensive Centers, on a technical assistance event to help 
their regions with American Recovery and Reinvestment Act (ARRA) issues. REL Southeast was 
responsible for bringing all the various entities together for the event. REL Southeast staff 
highlighted the following projects and activities occurring under this mission: 

• Native American Education Report. REL Southeast joined forces with other RELs to 
produce a report on Native American education. This report was led by REL Northwest. 

• National Reference Desk. RELs logged all constituent questions and responses into a 
REL- wide database referred to as the National Reference Desk. According to the REL 
director, this resource was helpful to all RELs because when constituents inquired about a 
particular subject, the REL was able to draw on the database to see if any REL had 
responded to a similar question. 

• REL Director “Leader.” The REL Southeast director was elected by the other REL 
directors to serve as the REL director “leader.” In this position, he organized phone calls 
and meetings of the various REL directors and served as a single point of contact for IES 
when it needed to quickly assemble all 10 directors/RELs. 

Mission #7: School Finance Systems 

REL Southeast staff highlighted the following project or activity occurring under this mission: 

• EBE Request Desk and Database. The REL director indicated that the EBE Request 
Desk and Database were used to respond to constituents’ needs in the area of school finance 
systems (among many other topics). As mentioned previously, the EBE Request Desk is a 
tool that was used by the REL to respond quickly to its constituents’ questions. In response 
to the requests, the REL produced formal “mini-reports,” written email responses, trainings, 
and/ or Bridge Events. 

Mission #8: Alternative Administrative Structures 

The REL director indicated that in addition to the activities listed below, the EBE Request Desk and 
Database were used to respond to constituents’ needs in the area of alternative administrative 
structures. The REL director said that the requests were in many areas, including block scheduling, 
the structure of SEA departments, and school-specific structures. REL Southeast staff highlighted 
the following projects and activities occurring under this mission: 

• Lever for Change: Southeast Region State Initiative to Improve High Schools (FRP). 

This report categorized recent state activities in high school reform into six “levers for 
change.” It then placed the reform discussion in the context of an evidence-based decision- 
making process. The report also provided examples of research on reform activities. 
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• State Policies and Procedures and Selected Local Implementation Practices in 
Response to Intervention in the Six Southeast Region States (FRP). This report 
discussed the structures that are in place at various SEAs in REL Southeast’s region and how 
the SEAs implemented Rtl. Particular emphasis was placed on the similarities and 
differences between the various states’ stmctures. 

• Preparing Elementary School Teachers in the Southeast Region to Work with 
Students with Disabilities (FRP). This report addressed how various states in REL 
Southeast’s region prepared teachers to work with students with disabilities. Like the above 
report, this FRP highlighted similarities and differences between states’ stmctures. 

Mission #9: School Improvement Plans 

REL Southeast staff highlighted the following projects and activities occurring under this mission: 

• EBE Request Desk Response Example # 1 . REL Southeast received a request from a 
state to help it review its school districts’ school improvement plans. The REL provided 
training for the state and helped the state develop effective ways to review the plans. 

• EBE Request Desk Response Example #2. A constituent requested information on the 
research base to support multiple components of a required plan for low-performing high 
schools, including ninth-grade transition, formative assessments, literacy issues and needs, 
assistance for struggling students, professional development based on data, professional 
learning community, processes and procedures stmctured to help all students achieve 
proficiency, community involvement, and comprehensive redesign/ reform. REL Southeast 
responded to this request by summarizing the available literature on these educational areas 
and submitting the document to the constituent. 

• EBE Request Desk Response Example #. 3 . REL Southeast responded to a constituent’s 
request for information on innovative practices for school reform with a mini report on the 
topic. The constituent sent this request to support work with School Improvement Grants. 

Mission #10: Technology in Education 

REL Southeast staff highlighted the following projects and activities occurring under this mission: 

• EBE Request Desk Response Example HI. In response to a constituent inquiry, the 
REL responded to the following questions: (1) Do other states certify online course 
providers for the K— 12 market? (2) Do state agencies provide guidelines for standards of 
online courses? (3) If not, what national organizations provide standards for quality online 
courses? 

• EBE Request Desk Response Example #2. The REL received a request and responded 
to the following questions: (1) What districts/ states are implementing technology on a 1-to-l 
ratio to students? (2) What have studies shown about the impact of this on student learning? 
(3) What have studies shown about the best use of the technology? 
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• EBE Request Desk Response Example #3. Through the EBE Request Desk, REL 
Southeast provided a response to the following questions asked by a constituent: (1) Are 
there school districts in other states that contract with private providers to do part-time 
virtual programs? (2) If so, what is the status of those programs? 

Quality and Relevance of Fast Response Projects 

As of December 1, 2009, 60 percent (12) of REL Southeast’s 20 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 11-1). Sixty percent (6) of REL Southeast’s 10 submissions for FRP reports 
were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 


By December 1, 2009, REL Southeast had six FRP reports accepted and published by IES: 

1 . Evidence-Based Decision-making: Assessing Reading Across the Curriculum Interventions (June 
2007); 

2. A Review of Methods and Instruments Used in State and Local School Readiness Evaluations 
(August 2007); 

3. Levers For Change: Southeast Region State Initiatives to Improve High Schools (September 2007); 

4. State Policies and Procedures and Selected Local Implementation Practices in Response to Intervention in 
the Six Southeast Region States (September 2008); 

5. Preparing Elementary School Teachers in the Southeast Region to Work with Students with Disabilities 
(November 2008); and 

6 . Reducing Stereotype Threat in Classrooms: A Review of the Social-Psychological Intervention Studies on 
Improving the Achievement of Black Students (June 2009). 

Table 11-2 shows the expert panel quality and relevance ratings for these IES-published reports 
from REL Southeast and for the six corresponding proposals. 73 Average quality and relevance 
ratings for proposals and reports from all 10 RELs are provided for reference. Quality rating 
categories ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged 
from 1 (“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and the 
corresponding IES-published reports received a mean quality rating of 3.81. FRP proposals from 


73 Appendix A describes in detail the indicators that make up the quality and relevance dimensions. 
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Table 11-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Southeast and all 10 RELs combined as of December 1, 2009 



Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 

Southeast 

20 

12 

60 

All RELs 

297 

137 

46 

Reports 

Southeast 

10 

6 

60 

All RELs 

166 

92 

55 


Table Reads : Of the 20 REL Southeast proposals submitted, 12 (60 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


REL Southeast received a mean quality rating of 3.23, and the corresponding IES-published reports 
received a mean quality rating of 3.61. Among the six REL Southeast proposals, mean quality ratings 
ranged from 2.39 to 3.69. Among the six REL Southeast IES-published reports, mean quality ratings 
ranged from 2.93 to 4.21 . 

With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 
mean rating of 3.64 for the actual reports. REL Southeast proposals received a mean relevance 
rating of 3.48, and the corresponding IES-published reports received a mean relevance rating of 
3.60. Among the six REL Southeast proposals, the mean relevance ratings ranged from 2.89 to 4.00. 
Among the six REL Southeast IES-published reports, the mean relevance ratings ranged from 3.12 
to 4.25. Additional detail on the distribution of the ratings is provided in Figure 11-1. 
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Table 11-2. Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Southeast and all 10 RELs combined 
(on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All REL FRPs (/V = 7 5) 

3.24 

3.39 

All Southeast FRPs (N = 6) 

3.23 

3.48 

A Review of Methods and Instruments Used in State and 
Local School Readiness Evaluations 

3.50 

4.00 

Evidence Based Decision-making: Assessing Reading 
Across the Curriculum Interventions 

3.56 

3.89 

Levers for Change: Southeast Region State Initiatives to 
Improve High Schools 

3.69 

3.44 

Preparing Elementary School Teachers in the Southeast 
Region to Work with Students with Disabilities 

3.33 

3.67 

Reducing Stereotype Threat in Classrooms: A Review of 
Social-Psychological Intervention Studies on Improving the 
Achievement of Black Students 

2.92 

3.00 

State Policies and Procedures and Selected Local 
Implementation Practices in Response to Intervention in 
the Six Southeast Region States 

2.39 

2.89 

Reports 


All REL FRPs (N = 91) 

3.81 

3.64 

All Southeast FRPs ( N = 6) 

3.61 

3.60 

A Review of Methods and Instruments Used in State and 
Local School Readiness Evaluations 

4.19 

3.88 

Evidence-Based Decision-making: Assessing Reading 
Across the Curriculum Interventions 

2.93 

3.12 

Levers for Change: Southeast Region State Initiatives to 
Improve High Schools 

3.17 

3.25 

Preparing Elementary School Teachers in the Southeast 
Region to Work with Students with Disabilities 

4.21 

4.25 

Reducing Stereotype Threat in Classrooms: A Review of 
Social-Psychological Intervention Studies on Improving the 
Achievement of Black Students 

3.81 

3.88 

State Policies and Procedures and Selected Local 
Implementation Practices in Response to Intervention in 
the Six Southeast Region States 

3.33 

3.25 


Table Reads: For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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Figure 11-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Southeast 
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Figure Reads: 5.4 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on 
a 5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 92 and 47 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 119 and 60 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 6 

N (Reports) = 6 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: Southwest 



Regional Educational Laboratory Southwest serves a region that includes the following states: 

• Arkansas; 

• Louisiana; 

• New Mexico; 

• Oklahoma; and 

• Texas. 

For the 2006-201 1 contract period, REL Southwest was headquartered at Edvance Research, Inc. in 
San Antonio, Texas. This was the first REL contract that Edvance Research had held. 

Description of Projects and Activities 

In May 2010, a member of the evaluation study team interviewed the REL Southwest director and 
six REL staff members. They were asked to provide up to three examples of the work their REL 
had completed under each mission specified in ESRA. 74 Other than documenting REL reports 
published by IES, the evaluation study team made no attempt to verify independently the reported 
information on REL activities, so the accuracy and completeness of this information is unknown. 

Mission #1: Training or Technical Assistance 

In addition to the examples of REL Southwest’s training and technical assistance work described 
below, the REL also completed a quick-turnaround request for a representative in the Texas state 
legislature, in which REL staff members cross-referenced recommendations in a bill under 
consideration in the legislature with empirical education research. REL Southwest staff highlighted 
the following projects and activities occurring under this mission: 

• TA Response Unit/Research READYSOURCE and Ask-A-REL. REL Southwest 
catalogued all constituents’ requests into its data management system and was able to track 


74 For a more detailed description of the methodology, see chapter 2. 
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the types of requests it received, by type of constituent. After receiving the requests and 
entering them into the database, the REL staff members responded to the inquiries with 
research references and information on the quality and quantity of existing research on a 
subject. 

• Technical Assistance Work with the Commission for College Ready Texas (CCRT). 75 

The REL Southwest director was invited to be on this commission and co-chair the research 
subcommittee. 

• Bridge Events. REL Southwest conducted a Bridge Event on ‘Understanding and 
Strengthening Response to Intervention (Rtl ) in Practice” in response to emerging needs in the 
region. This Bridge Event was designed to enhance participants’ understanding of how to 
identify and use evidence-based information. As a follow-up to the Bridge Event, REL 
Southwest staff members presented at a regional conference hosted by REL Southeast on 
using research to strengthen Rtl decision-making and implementation. In addition to the 
Bridge Event example above, REL Southwest, at the time of the interview, had held Bridge 
Events on seven additional topics: 

1. Assisting Students Struggling with Reading: Response to Intervention (Rtl) and 
Multi-Tier Intervention in the Primary Grades 

2. Educator Staffing, Quality, and Teacher Retirement Benefit Systems 

3. Understanding and Implementing Response to Intervention (Rtl) in New Mexico 

4. Texas Consortium on School Research (TCSR) 

5. Out-of-School Time and Student Achievement 

6. American Reinvestment and Recovery Act (ARRA) Event “Education Reform: 
Building Coherence with ARRA” 

7. Texas Consortium on School Readiness: College Readiness 

Mission #2: Dissemination 

REL Southwest used the results of its quantitative needs assessment, which according to the REL 
showed that awareness of the RELs was very low, to work with the U.S. Department of Education 
(ED) to develop a consistent report format that was engaging and could be used for all studies 
coming out of the REL system. The REL also commented that for two reasons it relied for 
dissemination more on its own website, rather than the IES website. First, the REL could quickly 
make updates. In the past, the REL staff members stated that they had problems with broken links 
and with a lack of responsiveness from the administrator of the IES REL website; however, this 
situation subsequently improved. Second, the REL was able to distribute a wider array of materials 


75 http:/ / www.collegereadytexas.org/ 
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to a broader audience through its own website. REL Southwest staff highlighted the following 
projects and activities occurring under this mission: 

• Customized Dissemination Products. REL Southwest customized the dissemination 
products based on the audience to make its products user friendly. In addition to longer 
reports, the REL produced press releases, short briefs, and newsletters for policymakers and 
other constituents. Most short products contained a link to the longer report. 

• Strategic Dissemination Campaigns. REL Southwest used press releases to highlight the 
involvement of specific districts and schools as well as to highlight studies as they were 
released. REL Southwest started the dissemination effort for Fast Response Projects (FRPs) 
with an announcement to alert the field that a report was forthcoming, followed by interim 
bulletins whenever possible. The REL also ran ads in education Week 76 that featured reports 
that it had released as well as other reports that it believed might interest the states in the 
region. These ads included links to the specific reports as well as a link to the REL system as 
a whole. The REL used emails and on-line distribution of the reports and other materials 
through its website. The REL also handed out hard copies of the reports at Bridge Events. 

• Monitoring of Dissemination Efforts. According to the REL staff members, REL 
Southwest tracked all dissemination efforts to evaluate their effectiveness. Through this 
tracking effort, for example, REL Southwest documented an increase in the number of 
visitors to the website after running an ad in Education Week. According to the REL director, 
at the time of the interview, the REL was in the process of upgrading its customer database 
so the REL would have more detailed information on how users access its website. 

Mission #3: Needs Assessment or Needs Sensing 

REL Southwest staff highlighted the following projects and activities occurring under this mission: 

• Survey of REL Southwest Constituents. The survey of constituents queried 1,212 
respondents from six groups within each of the five states in the REL’s region — the business 
community, higher education personnel, district superintendents, principals, teachers, and 
parents. The analysis provided information on the unique and common education research 
needs of the constituent base and allowed REL Southwest to examine the education research 
needs in the region, by state, and by constituent group. The REL reported that it used the 
results of the survey to inform its research agenda. 

• Individual, Group, and State Outreach. REL Southwest met with individuals in a variety 
of roles in education from each state in the region, including state chiefs in all five states, 
state science and math directors, and representatives of the teachers’ unions. REL 
Southwest, along with its Governing Board members, also met with members of Congress in 
Washington, DC, to talk about the work of REL Southwest. 


76 http:/ /www.edweek.org/ew/index.html 


Evaluation of the Regional Educational Laboratories: Interim Report 

117 


• Governing Board Meetings. The REL Southwest Governing Board met three times a year. 
In preparation for these meetings, REL Southwest staff members stated that the REL 
identified questions it thought could be addressed within each topic and then ranked the 
questions in a presentation to the board. The board used this information as input to help 
direct the work of the REL. 

Mission #4: Carrying Out Applied Research Projects 77 

REL Southwest staff highlighted the following projects and activities occurring under this mission: 

• Assessing the Impact of Collaborative Strategic Reading (CSR) on Reading 
Comprehension (Impact Study). This classroom-level random assignment study examined 
the impact of the CSR technique on students in grade 5 in Texas and Oklahoma. The study 
was in 26 schools and 74 classrooms and had control and treatment teachers within each 
school. The main research question for the study was “Does CSR lead to higher reading 
comprehension for English language learner (ELL) students versus non-ELL students?” ' 8 

• Effectiveness of Small Group Ma thema tic Intervention for Struggling First Graders 
(Impact Study). This school-level random assignment study gathered causal evidence on 
whether Rtl is effective across urban districts in four states in 37 treatment and 37 control 
schools. For this study, at-risk first-graders were pulled out of their regular classrooms in 
small groups three times per week for 40 minutes at a time. The students received 
approximately 30 hours of this separate instruction across the semester. The main research 
questions for the study were: (1) Is student performance in mathematics improved through 
this treatment? and (2) Was there a differential effect in student performance based on the 
severity of the children’s at-risk status? 79 

• How a Summ er R ea ding In terven tion Based on Lexiles Affects R ea ding 
Comprehension (Impact Study). This was a student-level random assignment study in 
which all students received the intervention. Students were randomly assigned into two 
groups. The treatment group received the treatment over the summer; the control group 
received the treatment in the fall, following the posttest. Struggling readers were provided 
with free books that were matched to them based both on their reading level and on topic 
areas the students said they were interested in. The main research question for the study was 
“Do students who receive the free books read more?” 81 

Mission #5: Providing Applied Research Projects in a Usable Format 

REL Southwest staff highlighted the following projects and activities occurring under this mission: 


77 Data have been collected for the three impact studies described under this mission and, at the time of the interview, the reports 
were being written. 

78 IES published a report from this study in March 2011. 

79 IES published a report from this study in February 2012. 

80 http:/ /www.lexile.com/about-lexile/lexile-codes/ 

81 IES published a report from this study in March 2012. 
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82 

Bridge Events for the Texas Consortium on School Research. ~ REL Southwest set up 
a consortium of 18 districts in Texas. As of spring 2010, the TCSR met twice and planned to 
hold a third meeting by webinar. The first meeting presented a user’s practice guide related 
to research on programs for college readiness and briefed the participants on the work of the 
Consortium on Chicago School Research. The second meeting was on improving readiness 
for college and careers and also included a practice guide. 


• Standards Alignment/ Crosswalks and Gap Analysis Issues (FRPs). This project 
encompassed a set of three alignment studies done as FRPs. The first looked at the 
alignment of the National Assessment of Educational Progress (NAEP) across all of the 
states in the REL region in math and science. The second examined four national 
assessments of college writing standards (ACT), American Diploma Project (ADP), and 
the College Board ) and compared them to ADP as the benchmark. The third project 
looked at the ACT College Readiness Standards in comparison to the English language arts 
and reading Texas Essential Knowledge and Skills curriculum standards. 


• Avoidable Losses (FRP). The Texas Education Agency (TEA) asked REL Southwest to 
review a report TEA had prepared on summer reading loss to assess the soundness of the 
study methodology and the appropriateness of the conclusions drawn in the report. REL 
Southwest described this report as a translation of technical assistance into a written 
document that could be disseminated and reviewed by IES’ Analytical and Technical 
Support contractor. 

Mission #6: Coordination or Collaboration 


REL Southwest highlighted the three activities described below to illustrate its collaboration with 
other RELs and Comprehensive Centers, but also discussed several other examples of collaboration, 
including a Native American study several RELs had been involved in and an upcoming study led by 
REL Mid-Atlantic on longitudinal data systems. REL Southwest staff highlighted the following 
projects and activities occurring under this mission: 

• Bridge Event on the American Recovery and Reinvestment Act (ARRA). REL 

Southwest worked with REL West and REL Central, as well as several of the 
Comprehensive Centers to hold a forum to help states develop a better understanding of 
ARRA. According to the REL, in addition to providing support to the forum, REL 
Southwest did all of the signage and branding to highlight the REL presence. 


82 http://tcsr.edvanceresearch.com/ 

83 http:/ / ccsr.uchicago.edu/ content/index.php 

84 http:/ /nces. ed.gov/nationsreportcard/ 

85 http:/ Avww.act.org/ 

86 http:/ / www.achieve.org/adp-network 

87 http:/ / www.collegeboard.org/ 

88 http:/ /www.tea.state.tx.us/index2.aspx?id=6148 
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• National Laboratory Network. REL Southwest cited much of the work it does under Task 
3 (i.e., NLN) as examples of coordination or collaboration. This includes the work the REL 
did at the start of the contract on REL branding, collaborations on the REL Intranet and 
Internet, participation in the working groups, and the NCES Management Information 
Systems Conference. 

Mission #7: School Finance Systems 

REL Southwest staff highlighted the following projects and activities occurring under this mission: 

• Bridge Event on Educator Staffing, Quality, and the Teacher Retirement Benetit 
systems. REL Southwest hosted an event to provide information about the current 
landscape of teacher retirement systems and facilitate an opportunity for state policymakers 
to discuss possibilities for using retirement benefit systems to improve teacher quality, 
specifically recmitment and retention. 

• Examining Context and Challenges in Measuring Investment in Professional 
Development: A Case Study of Six School Districts in the SW Region (FRP). REL 

Southwest worked with six districts in four states (two districts in Arkansas, two in New 
Mexico, one in Oklahoma, and one in Texas) to identify data sources to determine how 
much these districts were spending on teacher and principal professional development 
activities. According to the REL, the resulting analysis illustrated some of the challenges 
associated with measuring investment in professional development and highlighted online 
data systems as a way to potentially improve accuracy of spending estimates and expenditure 
tracking. 

Mission #8: Alternative Administrative Structures 

REL Southwest interpreted this mission primarily to refer to work done in the area of charter 
schools and highlighted two of its activities in this area. REL Southwest staff highlighted the 
following projects and activities occurring under this mission: 

• Technical Assistance Work with Charter Schools. REL Southwest highlighted several 
examples in which it had provided technical assistance in the area of charter schools. The 
REL provided the Knowledge Is Power Program (KIPP) Foundation with guidance and 
review around the design of the National Study on KIPP’s Impact at the middle-school level 
and helped to select the contractor for the evaluation. REL Southwest staff members 
reported on the progress of the Louisiana Recovery School District 90 and on the first year 
eStem Public Charter School 91 to its governing board. 

• Conference on Five Years of Post-Katrina School Reform. At the time of the interview, 
REL Southwest was planning to co-sponsor an event with the University of New Orleans 


89 http: / / www.kipp.org/ 

90 http:/ /www.rsdla.net/ 

91 https://www.estemlr.net/ 
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focusing on education research and school reform taking place in the storm-affected 
communities, with particular emphasis on the role of charter schools. 

Mission #9: School Improvement Plans 

REL Southwest staff highlighted the following projects and activities occurring under this mission: 

• Technical Assistance for the TCSR. As was discussed under Mission #5, the REL saw 
this activity as a mechanism to connect IES research findings with the real world of practice 
in school districts through increasing districts’ awareness and usage of research and data, 
providing a forum for districts to interact with research experts, providing an opportunity 
for district leaders to engage with each other, and encouraging districts to more fully utilize 
research in their decision-making process. 

• Request from Texas State Legislator. As was mentioned under Mission #1, REL 
Southwest provided research to the legislative representative on improving failing schools. 
The REL then shared the materials it created through this effort with the regional 
educational service centers 9 " in Texas that led the turnaround effort for the schools/ districts 
in the region. 

• Bridge Event and Forum on Out-of-School Time (OST) and Student Achievement. 

REL Southwest presented information regarding OST and the research supporting its 
impact on student achievement, considerations for evaluating OST programs, and 
recommendations for implementing an evidence-supported multi-tiered model of 
instruction. The IES publication Structuring Out-of-School Time to Improve Academic Achievement: 
A Practice Guide was part of the presentation. This work was a collaboration between REL 
Southwest and the Texas Comprehensive Center. 9 ’ 

Mission #10: Technology in Education 

REL Southwest staff highlighted the following projects and activities occurring under this mission: 

• Bridge Event for the Texas Consortium on School Research, Introduction to State 
Tools for Teaching Excellence. Although mentioned earlier under Missions #5 and #9, 
REL Southwest included this activity under Mission #10 to highlight the community of 
practice (e.g., an online forum to share knowledge) it developed using technology to support 
the users. The REL used the community of practice to help the districts access their data 
once the REL completed the analyses for its on-track for high school graduation indicators. 

• Administrative Data Use of the On-Track for High School Graduation Indicator for 
Texas. This study examined the effectiveness of using the CCSR on-track indicator in five 
school districts in Texas. According to the REL, through this tool, educators can use 
indicators to analyze which students are having trouble in particular areas and identify which 
research-based practices are most successful with those students. The goal was to help 


92 http:/ /www. texasresc.net/ default.htm 

93 http:/ / txcc.sedl.org/ 
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educators easily access information about research-based interventions that are connected to 
the persistent problems students are experiencing. 

• Bridge Event on Using Student Achievement Data, to Support Instructional 
Decision-Making . REL Southwest presented this practice guide at the meeting of the 
Texas Association of School Administrators 94 . This Bridge Event revolved around using 
student achievement data to improve student outcomes. 

Quality and Relevance of Fast Response Projects 


As of December 1, 2009, 49 percent (21) of REL Southwest’s 43 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 12-1). Seventy-seven percent (17) of REL Southwest’s 22 submissions for 
FRP reports were accepted for publication by IES, compared to 55 percent for all 10 RELs 
combined. 

By December 1, 2009, REL Southwest had 17 FRP reports accepted and published by IES: 

1 . How the Government Defines Rural Has Implications for Education Policies and Practices (June 2007); 

2. Aligning Science Assessment Standards: Arkansas and the 2009 National Assessment of Educational 
Progress (NAEP) (July 2007); 

3. Aligning Science Assessment Standards: Louisiana and the 2009 National Assessment of Educational 
Progress (NAEP) (July 2007); 


Table 12-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
Southwest and all 10 RELs combined as of December 1, 2009 



Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 

Southwest 

43 

21 

49 

All RELs 

297 

137 

46 

Reports 

Southwest 

22 

17 

77 

All RELs 

166 

92 

55 


Table Reads: Of the 43 REL Southwest proposals submitted, 21 (49 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


94 http://www.tasanet.org/ 


Evaluation of the Regional Educational Laboratories: Interim Report 

122 


4. Aligning Science Assessment Standards: New Mexico and the 2009 National Assessment of Educational 
Progress (NAEP) (July 2007); 

5. Aligning Science Assessment Standards: Oklahoma and the 2009 National Assessment of Educational 
Progress (NAEP) (July 2007); 

6. Aligning Science Assessment Standards: Texas and the 2009 National Assessment of Educational 
Progress (NAEP) (July 2007); 

7. Examining Context and Challenges in Measuring Investment in Professional Development: A Case Study 
of Six School Districts in the Southwest Region (September 2008); 

8. Aligning Mathematics Assessment Standards: Arkansas and the 2009 National Assessment of 
Educational Progress (NAEP) (October 2008); 

9. Aligning Mathematics Assessment Standards: Eouisiana and the 2009 National Assessment of 
Educational Progress (NAEP) (October 2008); 

10. Aligning Mathematics Assessment Standards: New Mexico and the 2009 National Assessment of 
Educational Progress (NAEP) (October 2008); 

1 1 . Aligning Mathematics Assessment Standards: Oklahoma and the 2009 National Assessment of 
Educational Progress (NAEP) (October 2008); 

12. Aligning Mathematics Assessment Standards: Texas and the 2009 National Assessment of Educational 
Progress (NAEP) (October 2008); 

13. Ea Frontera: Student Achievement in Texas and Border and Nonborder Districts (September 2007); 

14. Previewing the Evidence on How Teacher Professional Development Affects Student Achievement (October 
2007); 

15. Formative Assessment Policies, Programs, and Practices in the Southwest Region (January 2008); 

16. Alternate Assessments for Special Education Students in the Southwest Region States (February 2008); 
and 

17. A Review of Avoidable Fosses: High Stakes Accountability and the Dropout Crisis (February 2008). 

Table 12-2 shows the expert panel quality and relevance ratings for these IES-published reports 
from REL Southwest and for the eight corresponding proposals. 95 Average quality and relevance 
ratings for proposals and reports from all 10 RELs are provided for reference. Quality rating 
categories ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged 
from 1 (“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 


95 Two proposals resulted in 10 reports, and 1 report was without a corresponding proposal, as noted in Table 12-2. Appendix A 
describes in detail the indicators that make up the quality and relevance dimensions. 
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For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and the 
corresponding IES-published reports received a mean quality rating of 3.81. FRP proposals from 
REL Southwest received a mean quality rating of 2.96, and the corresponding IES-published reports 
received a mean quality rating of 3.81. Among the eight REL Southwest proposals, mean quality 
ratings ranged from 2.06 to 4.19. Among the 17 REL Southwest IES-published reports, mean 
quality ratings ranged from 2.47 to 4.46. 


Table 12-2. Expert panel quality and relevance ratings for IES-published Fast Response Project 
reports and corresponding proposals from REL Southwest and all 10 RELs 
combined (on a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All RELFRPs (N = 75) 

3.24 

3.39 

All Southwest FRPs (/V = 8) 

2.96 

3.33 

Aligning mathematics assessment standards: Arkansas 
[and Louisiana and New Mexico and Oklahoma and Texas] 
and the 2009 National Assessment of Educational 
Progress (NAEP) 

3.75 

3.22 

Aligning science standards: Arkansas [and Louisiana and 
New Mexico and Oklahoma and Texas] and the 2009 
National Assessment of Educational Progress (NAEP) 

4.19 

4.33 

Alternate Assessments for Special Education Students in 
the Southwest Region States 

3.00 

3.44 

Examining Context and Challenges in Measuring 
Investment in Professional Development: A Case Study of 
Six School Districts in the Southwest Region 

2.44 

3.22 

Formative Assessment Policies, Programs, and Practices 
in the Southwest Region 

2.13 

2.75 

How the Government Defines Rural Has Implications for 
Education Policies and Practices 

2.93 

3.75 

La Frontera: Student Achievement In Texas and Border 
and Nonborder Districts 

2.06 

2.67 

Reviewing the Evidence On How Teacher Professional 
Development Affects Student Achievement 

3.18 

3.22 

Reports 


All REL FRPs (N = 91) 3.81 

3.64 

All Southwest FRPs (N = 17) 3.81 

3.60 
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Table 12-2. Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL Southwest and all 10 RELs 
combined (on a 5-point scale with 5 being the highest) (continued) 


Mean Ratings 

Quality 

Relevance 

A Review of Avoidable Losses: High Stakes Accountability and 
the Dropout Crisis(-) 

4.14 

4.25 

Aligning mathematics assessment standards: Arkansas and 
the 2009 National Assessment of Educational Progress (NAEP) 

3.47 

2.90 

Aligning mathematics assessment standards: Louisiana and 
the 2009 National Assessment of Educational Progress (NAEP) 

3.47 

2.90 

Aligning mathematics assessment standards: New Mexico and 
the 2009 National Assessment of Educational Progress (NAEP) 

3.47 

2.90 

Aligning mathematics assessment standards: Oklahoma and 
the 2009 National Assessment of Educational Progress (NAEP) 

3.47 

2.90 

Aligning mathematics assessment standards: Texas and the 
2009 National Assessment of Educational Progress (NAEP) 

3.47 

2.90 

Aligning science assessment standards: Louisiana and the 
2009 National Assessment of Educational Progress (NAEP) 

4.30 

4.15 

Aligning science assessment standards: New Mexico and the 
National Assessment of Educational Progress (NAEP) 

4.30 

4.15 

Aligning science assessment standards: Oklahoma and the 
2009 National Assessment of Educational Progress (NAEP) 

4.30 

4.15 

Aligning science assessment standards: Texas and the 2009 
National Assessment of Educational Progress (NAEP) 

4.30 

4.15 

Aligning science standards: Arkansas and the 2009 National 
Assessment of Educational Progress (NAEP) 

4.30 

4.15 

Alternate Assessments for Special Education Students in the 
Southwest Region States 

4.46 

4.50 

Examining Context and Challenges in Measuring Investment in 
Professional Development: A Case Study of Six School Districts 
in the Southwest Region 

3.35 

3.42 

Formative Assessment Policies, Programs, and Practices in 
the Southwest Region 

4.25 

3.83 

How the Government Defines Rural Has Implications for 
Education Policies and Practices 

2.47 

3.00 

La Frontera: Student Achievement In Texas and Border and 
Nonborder Districts 

3.62 

3.25 

Reviewing the Evidence On How Teacher Professional 
Development Affects Student Achievement 

3.52 

3.67 


Table Reads : For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. (-) = A proposal was not available for the report, “A Review of Avoidable Losses: High Stakes Accountability and the Dropout 
Crisis.” 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 
mean rating of 3.64 on for the actual reports. REL Southwest proposals received a mean relevance 
rating of 3.33, and the corresponding IES-published reports received a mean relevance rating of 
3.60. Among the eight REL Southwest proposals, the mean relevance ratings ranged from 2.67 to 
4.33. Among the 17 REL Southwest IES-published reports, the mean relevance ratings ranged from 
2.90 to 4.50. Additional detail on the distribution of expert panelist indicator-level quality and 
relevance ratings is provided in Figure 12-1. 


Figure 12-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
Southwest 
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Figure Reads: 8.5 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on 
a 5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 130 and 70 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 369 and 204 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 8 

N (Reports) = 17 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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REL: West 


13 


Regional Educational Laboratory West serves a region that includes the following states: 

• Arizona; 

• California; 

• Nevada; and 

• Utah. 

For the 2006-201 1 contract period, REL West was headquartered at WestEd in San Francisco, 
California. WestEd had held previous REL contracts. 

Description of Projects and Activities 


In June 2010, a member of the evaluation study team interviewed the REL West director and 
7 members of the REL’s staff. REL staff members were asked to provide up to three examples of 
the work their REL had completed under each mission specified in ESRA. % Other than 
documenting REL reports published by IES, no attempts were made by the evaluation study team to 
verify independently the reported information on REL activities, so the accuracy and completeness 
of this information is unknown. 

Mission #1: Training or Technical Assistance 

REL West staff highlighted the following projects and activities occurring under this mission: 

• Analysis of the Utah K-3 Reading Improvement Program (Fast Response Project / 

FRP). REL West prepared this report at the request of the Utah state superintendent. Utah 
was in Year 2 of a pilot program of the K— 3 Reading Improvement Program within the 
state. The state wanted the following questions answered: (1) Who in the state had 
implemented the program? (2) How was the program being implemented? (3) What was the 
impact of the program, if any? REL West presented the descriptive data to the state 
legislature and state board. The REL staff members stated that they also used this 


96 For a more detailed description of the methodology, see chapter 2. 
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opportunity to work with the Utah SEA to help build its research literacy and understanding 
of evaluation design so that more rigorous impact analyses could be conducted in the future. 

• Los Angeles (LA ) UniGed School District (USD) Partnership Projects. The LA USD 

created a task force on teacher effectiveness in September 2009 in response to a study that 
found that it was rare for a teacher in the district to be dismissed for performance. The task 
force met several times over the course of six months and made a series of 
recommendations to the school board. The district personnel then asked REL West to help 
them think through the task force recommendations. As of June 2010, REL West had 
completed or planned several projects to help the district, such as three completed Reference 
Desk memos — a review of district professional development practices, an overview of 
descriptive information on teacher leader positions in other districts, and options for 
measuring teacher effectiveness; an upcoming Bridge Event to engage teachers and 
principals in a discussion of research on the attributes of effective teaching; and plans to 
participate in the emerging LA Research Consortium (a group of universities in the LA area 
working with REL West) to help pilot test different teacher effectiveness measures. 

• Bridge Events. The Nevada legislature discussed developing college readiness standards for 
the state, but given the economic conditions in the state, the legislature was unable to fund 
the initiative. According to REL staff members, college readiness standards had been a 
prominent issue in the region for the last several years. State agencies worked together to 
launch a task force even without funding. The REL, in collaboration with the Nevada state 
superintendent and the vice chancellor of the Nevada system of higher education, presented 
a Bridge Event on pathways to college to inform the task force. The presenters used the IES 
practice guide on pathways to college and worked with event attendees to understand 
Nevada’s status in the process of implementing the practice guide recommendations. REL 
West reported that the event had approximately 100 attendees, including representatives 
from all 17 school districts in the state, as well as representatives from the community 
college and university systems. The first kickoff meeting for the task force took place the 
same day as the Bridge Event. The task force continued to meet throughout that winter and 
developed recommendations for readiness standards, which were presented to the legislature 
in late spring 2010. In addition to conducting the Bridge Event described above, REL West, 
at the time of the interview, had held Bridge Events on five additional topics: 

1. Turning Around Chronically Low Performing Schools: Evidence-Based Practices; 

2. Turning Around Chronically Low Performing Schools: Implications for California’s 
SAIT Providers; 

3. Measuring Teacher Effectiveness: Implications for California’s Race to the Top; 

4. Turning Around Low Performing Schools: Bridging Research, Policy, and Practice; 
and 

5. Adolescent Uteracy: Effective Classroom and Intervention Practices. 
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Mission #2: Dissemination 

REL West engaged in multiple forms of dissemination with its constituents. For example, the REL 
disseminated press releases, published a research digest, sent out emails, and highlighted new reports 
on a portion of the WestEd website devoted to REL West. The REL also gave presentations to 
formal audiences such as the California Educational Research Association. REL West staff 
highlighted the following projects and activities occurring under this mission: 

• Reenrollment ofHS Dropouts (FRF ). REL West initiated this study in the San Bernardino 
(California) Unified School District after determining that little was known about what 
happened to students after they dropped out of school. As a result of the report, the San 
Bernardino district planned to implement a summer program as well as a rapid course-credit- 
recovery program to make it easier for dropouts to return and complete their remaining credits 
to graduate. REL West, in collaboration with a San Bernardino school official, did several 
presentations, including a webinar, of the FRP. 

• Trends in Teacher Demand (FRP). REL West published a study that analyzed 
projections for teacher demand in California at the county level and disseminated the 
information through higher education networks and in-person presentations. Then REL 
West initiated a tracer study (with fewer than nine respondents), to better understand how 
people used the information in this report. 

• Bridge Event on Measuring Teacher Effectiveness: Implications for California ’s 
Race to the Top (RTT). REL West worked with representatives from the California SEA 
who wanted to know more about measuring teacher effectiveness. For this event, REL West 
used materials from the Center for Teacher Quality % and brought in key researchers to talk 
to staff members working on California’s RTT plan. 

Mission #3: Needs Assessment or Needs Sensing 

In addition to the examples listed below, REL West created a review committee to help prioritize 
ways in which the REL addressed the needs in the region. This committee included REL West staff 
members, the state liaisons for each state, and content area experts as needed. In addition, the REL 
conducted a formal needs survey in the third year of this contract to determine whether the needs 
they had identified were aligned with what teachers, principals, and district administrators were 
reporting. REL West staff highlighted the following projects and activities occurring under this 
mission: 

• Example of Needs Analysis from the Year 4 Updated Annual Plan (UAP). This effort, 
summarized from REL West’s Year 4 UAP, included five components: (1) a systematic 


97 http:/ / cera-web.org/ 

98 http://www.calstate.edu/teacherquality/ 
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outreach to key groups, (2) a regular analysis of data and policy trends, (3) constituent 
outreach for targeting Bridge Events, (4) the development of specific state and local projects, 
and (5) the dissemination of current studies. 

• Needs Identification and Analysis System. REL West’s needs identification system, as 
reported by REL staff members, included systematic outreach to key groups through state 
board meetings; frequent contact with the legislature; and conversations with key leaders at 
the SEAs, the governor’s offices, and other stakeholders. The REL staff members also noted 
that the REL routinely analyzed policy reports and similar documents to identify needs. 
Although the REL tried using other forms of needs assessment, such as surveys, the REL 
staff members stated that they found that in-person, frequent contact to be the most 
effective way to identify needs in the region. 

• Summary Table of Needs in the Western Region. REL West produced a matrix that 
showed eight identified areas of need in the western region: (1) school and district 
improvement; (2) teacher and administrator quality; (3) educating English learners; (4) special 
education; (5) assessment, data use, and accountability; (6) readiness and community; (7) 
secondary school reform and higher education; and (8) school finances. The matrix also 
documented how each REL product linked to the identified need(s). 

Mission #4: Carrying Out Applied Research Projects 99 

REL West staff highlighted the following projects and activities occurring under this mission: 

• Experimental Study of the Program for Infant/Toddler Caregivers (Impact Study). 

This study examined an early childhood program that provided professional development 
for caregivers in both childcare centers and home-based childcare. REL West reported that 
this was the only study in the REL system that focused on early childhood. Ninety center- 
based sites and 160 family-based sites were randomly assigned to the treatment or control 
groups. The measured outcomes examined childcare quality as well as child outcomes, 
determined with in-home assessments. 1 "" 

• Experimental Study of Lessons in Character (Impact Study). This study evaluated a 
supplemental character education curriculum in elementary schools designed to introduce 
character education themes into the English language arts curriculum. There were 1 9 
supplemental 30-minute lessons that teachers delivered during the academic year. Fifty 
schools were in the study, which used school-level random assignment. Schools in the 
treatment group implemented the program in the second through fifth grades for 2 years. 
The outcome measures were academic achievement based on standardized test scores, 
teacher reports of social skills, and an assessment of school climate from interviews with 
principals and teachers. " 


99 For all three of the impact studies described below, REL West reported being in the final stages of data collection. 

100 IES published a report from this study in March 2012. 

101 IES published a report from this study in March 2012. 
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• Quality Teaching for English Learners (QTEL) (Impact Study). This study used a 
multiple-cohort, 3-year longitudinal design to assess the impacts of the QTEL" 12 teacher 
professional development program on teacher, classroom, and student outcomes. The 
QTEL program aimed to equip secondary teachers to provide challenging tasks and scaffold 
student learning to advance development of academic English fluency among ELL 
students. 103 

Mission #5: Providing Applied Research Projects in a Usable Format 

REL West staff highlighted the following projects and activities occurring under this mission: 

• Report Synopsis and Forum Presentation. REL West provided several examples of REL 
reports that it edited down to a two-page summary combined with a poster. In all cases, the 
abridged materials referenced the full report. REL West presented the materials at seminars 
and forums. Staff members reported that these presentations were helpful in that they 
spurred discussions. 

• Technical Assistance Briefing Packet. REL West conducted an analysis for officials in 
Fresno, California, that provided a current description of the enrollment status of young 
children in foster care in early intervention services and licensed preschools. The materials 
REL West produced were used in the May 2010 presentation to Fresno County officials and 
served as a basis for subsequent discussions. 

• Reference Desk Memos on Multiple Topics. REL West used the Reference Desk as a 
tool to respond to constituent inquiries. Some responses took only a few days to prepare, 
while others took up to 4 weeks. REL West produced multiple products in response to 
Reference Desk questions, such as a memo on college- and work-readiness indicators, a 
literature review on identifying budgetary savings within school districts, and a bibliography 
on education issues related to Native American students. 

Mission #6: Coordination or Collaboration 

In addition to the examples cited below, REL West participated in the National Laboratory Network 

(NLN) website, the Reference Desk, and Ask-A-REL. REL West staff highlighted the following 

projects and activities occurring under this mission: 

• Regional Forum on American Recovery and Reinvestment Act (ARRA) Funding. In 

November 2009, the RELs and the Comprehensive Centers had a conference call. As a 
result of this call, several RELs and Comprehensive Centers in collaboration with 13 to 14 
states organized a conference in Arizona on December 18, 2009. At the conference, federal 
employees presented information and talked with administrators from states and districts 
about ARRA and the expectations for the Race to the Top applications. 


102 http://ies.ed.gov/ncee/edlabs/projects/project.aspPProjectID— 88 

103 IES published a report from this study in March 2012. 
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• Bridge Event on Adolescent Literacy: Effective Classroom and Intervention 
Practices. This work was done in partnership with the California Comprehensive Center 104 
and a network of districts and counties in California that have taken on the mission of 
secondary literacy. At the Bridge Event, REL West presented research findings and 
facilitated discussions. 

• Bridge Event on High School Reform in the Southwest. REL West worked on high 
school reform in collaboration with the southwest states and the Southwest Comprehensive 
Center. 105 As a follow-up to the event, the REL participated in a conference on high school 
reform in May 2010. At the time of the interview, the REL anticipated that there would be 
additional follow-up on this activity in the future. 

Mission #7: School Finance Systems 

REL West staff highlighted the following projects and activities occurring under this mission: 

• Resource Utilization in Rural School Districts (FRP). This report examined variations 
in staffing patterns and resource utilization in rural districts across the states in REL West’s 
region. At the time of the interview, the REL staff members stated that they had been 
working on this brief for the past 14 months and had submitted the most recent draft in May 
20 10. 106 

• Implementation of a Weighted Student Formula in San Francisco (FRP). REL West 
was asked to describe the implementation and implications of the weighted student funding 
formula that San Francisco implemented in 2004. The REL staff members stated that this 
formula was very complex in its initial implementation and continued to evolve in the 
subsequent years to include weights associated with poverty, English learners, special 
education, and grade span. Unlike some of the other work REL West had done in this area, 
this was a case study and analysis of a particular district’s implementation of a policy. 

• Reference Desk Response on School-Based Budgeting and Management, Utah 
Request and Memo. The REL West state liaison, in collaboration with an official from the 
San Francisco Unified School District, presented the Reference Desk memo on school- 
based budgeting and management and described to the Utah legislature what the San 
Francisco district had done in this area. 

Mission #8: Alternative Administrative Structures 

REL West staff highlighted the following projects and activities occurring under this mission: 

• Examining Independent Study High Schools (FRP). REL West prepared this report in 
response to a request from the California Department of Education. The California 
administrator reported that the department was trying to manage, govern, and set standards 


104 http:/ / www.cacompcenter.org/ 

105 http://www.swcompcenter.org/cs/ swcc/print/htdocs/swcc/home.htm 

106 IES published a report from this study injanuary 2011. 
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for independent study high schools (a mix of Charter Schools and schools offering on-line 
study). This study addressed the following research questions: How are these schools 
organized? How are these schools governed? What is the population that attends these 
schools? REL West analyzed data from the California Department of Education and the 
public domain to produce a report that examined the trend in enrollment over a 5- to 6-year 
period from early 2000 through the 2006-07 school year and described examples of the 
governance structures in place. According to REL staff members, the report showed an 
increase in both the number and diversity of students in these types of schools. 

• Reference Desk Request to Review Teacher Leader Positions. Described under 
Mission #4, this work was part of the broader work REL West did with Los Angeles Unified 
School District. REL West provided information to the district on leadership pathways in 
four school districts. 

• Reference Desk Request on Optimal District Size. This work was done in response to a 
question from the state superintendent in Utah about the size of districts. REL West 
summarized the existing literature on district size that was used within Utah to feed into a 
larger debate in the legislature, trying to determine if it could save money by either breaking 
up large districts or consolidating smaller ones. 

Mission #9: School Improvement Plans 

REL West staff highlighted the following projects and activities occurring under this mission: 

• Bridge Events on Low-Performing Schools (LPS) for School Support Providers. This 
series of events brought together groups of school support providers from the region. REL 
West used the IES practice guide as the anchor and then had examples from the specific 
states. 

• Bridge Event on LPS for State Policymakers. REL West put together this Bridge Event 
in the context of California’s Race to the Top application to help the policymakers 
understand research findings on school turnaround. 

• Bridge Events on LPS for District and School Teams. At the time of the interview, REL 
West staff members reported that they were planning a series of Bridge Events targeted at 
district- and school-level leaders. The REL planned to focus these events on the district role 
in supporting school turnaround. The REL staff members stated that they hoped these 
events would allow participants to engage in some of the research and also leave with tools 
for implementing some of the research strategies. These Bridge Events were planned for fall 
2010 in collaboration with the California and Southwest Comprehensive Centers. 

Mission #10: Technology in Education 

REL West staff highlighted the following projects and activities occurring under this mission: 

• Reference Desk Memos on Computer-Adaptive Testing and Instruction. Utah asked 
REL West for a policy brief on implementing computer adaptive testing. In response, REL 
West put together information on ways in which particular states were using this form of 
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testing, lessons the states had learned, and a terminology glossary. The Utah state office of 
education used this memo to brief its legislature. 

• National Laboratory Network Resources. REL West reported that it had staff members 
download information from the NLN website. The REL staff members stated that they 
found certain aspects of the website more useful than others. For example, they said that the 
archive of the Thursdays @ 3 calls was quite useful. 

• Doing What Works Resource. This resource was developed at WestEd in partnership with 
the American Institutes for Research and RMC Research Corporation and served as a 
companion website to the What Works Clearinghouse 109 and the practice guides produced 
by IES. It took the information from the practice guides and communicated it to the public 
in a variety of ways through videotaped interviews with the panelists who created the guides 
and other experts. Doing What Works also featured a See How it Works section that 
includes some classroom videos and additional tools. REL West presented the Doing What 
Works in a 30-minute session as part of the larger Turning Around Low-Performing Schools 
Bridge Event in March 2009. 

Quality and Relevance of Fast Response Projects 


As of December 1, 2009, 67 percent (22) of REL West’s 33 FRP proposals were accepted by 
NCEERA and approved to continue as IES-supported projects, compared to 46 percent for all 10 
RELs combined (Table 13-1). Eighty percent (16) of REL West’s 20 submissions for FRP reports 
were accepted for publication by IES, compared to 55 percent for all 10 RELs combined. 


Table 13-1. Percentage of Fast Response Project proposals and reports accepted by IES for REL 
West and all 10 RELs combined as of December 1, 2009 



Number 

submitted 

Number 

accepted 

Percentage 

accepted 

Proposals 

West 

33 

22 

67 

All RELs 

297 

137 

46 

Reports 

West 

20 

16 

80 

All RELs 

166 

92 

55 


Table Reads: Of the 33 REL West proposals submitted, 22 (67 percent) were accepted. 
SOURCE: U.S. Department of Education, NCEERA. 


1117 http://www.air.org/ 

108 http://www.rmcresearchcorporation.com/ 

109 http://ies.ed.gov/ncee/wwc/ 
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By December 1, 2009, REL West had 15 FRP reports accepted and published by IES 11 ": 


1 . An Analysis of Utah ’s K—3 Reading Improvement Program (J une 2007) ; 

2. Measuring Resilience and Youth Development: The Psychometric Properties of the Healthy Kids Survey 
(September 2007); 

3. The Distribution of Teaching and Teaming Resources in California ’s Middle and High Schools 
(September 2007); 

4. Course-taking Patterns and Preparation for Postsecondary Education in California’s Public University 
Systems Among Minority Youth (January 2008); 

5. Characteristics of Arizona School Districts in Improvement (July 2008); 

6. Characteristics of California School Districts in Improvement (July 2008); 

7. Examining the Links Between Grade 12 Mathematics Coursework and Mathematics Remediation in 
Nevada Public Colleges and Universities (July 2008); 

8. The Reenrollment of High School Dropouts in a Large, Urban School District (July 2008); 

9. Trends in California Teacher Demand: A County and Regional Perspective (July 2008); 

1 0. Implementation of the Weighted Student Eormula Policy in San Francisco: A Descriptive Study of an 
Equity-Driven, Student-Based Planning and Budgeting Policy (August 2008); 

11. Training Early Intervention Assistants in California’s Community Colleges (September 2008); 

12. Characteristics of California School Districts in Program Improvement: 2008 Update (October 2008); 

13. Examining Independent Study High Schools in California (June 2009); 

14. A Multistate Review of Professional Teaching Standards (July 2009); and 

15. The Status of State-Level Response to Intervention Policies and Procedures in the West Region States and 
Five Other States (August 2009). 

Table 13-2 shows the expert panel quality and relevance ratings for these IES-published reports 
from REL West and for the 1 2 corresponding proposals. 1 1 1 Average quality and relevance ratings 


110 The number of FRP reports from REL West published by December 1, 2009, is one less than the number of submissions for FRP 
reports approved by that date, since one report — School-site administrators: a California county and regional perspective on labor market trends — 
was not published by IES until January 2010. This report was no! included in the expert panel reviews of IES-published FRP 
reports. 

111 One proposal resulted in three reports, and one report was without a corresponding proposal, as noted in Table 13-2. Appendix A 
describes in detail the indicators that make up the quality and relevance dimensions. 
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for proposals and reports from all 10 RELs are provided for reference. Quality rating categories 
ranged from 1 (“very weak”) to 5 (“very strong”), and relevance rating categories ranged from 1 
(“not relevant”) to 5 (“highly relevant”). Ratings of 3 were labeled “adequate.” 

For all 10 RELs, proposals for Fast Response Projects resulting in IES-published reports approved 
by December 1, 2009, received a mean quality rating of 3.24 on a 5-point scale, and the 
corresponding IES-published reports received a mean quality rating of 3.81. FRP proposals from 
REL West received a mean quality rating of 3.57, and the corresponding IES-published reports 
received a mean quality rating of 4.14. Among the 12 REL West proposals, mean quality ratings 
ranged from 2.17 to 4.39. Among the 15 individual REL West IES-published reports, mean quality 
ratings ranged from 3.58 to 4.57. 

With respect to relevance, all 10 RELs combined received a mean rating of 3.39 on a 5-point scale 
for proposals resulting in IES-published Fast Response Project reports by December 1, 2009, and a 
mean rating of 3.64 for the actual reports. REL West proposals received a mean relevance rating of 
3.42. and the corresponding IES-published reports received a mean relevance rating of 3.84. Among 
the 1 2 REL West proposals, the mean relevance ratings ranged from 3.00 to 4. 1 1 . Among the 1 5 
REL West IES-published reports, the mean relevance ratings ranged from 2.50 to 4.50. Additional 
detail on the distribution of expert panelist indicator-level quality and relevance ratings is provided in 
Figure 13-1. 
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Table 13-2. Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL West and all 10 RELs combined (on 
a 5-point scale with 5 being the highest) 


Mean Ratings 

Quality 

Relevance 

Proposals for corresponding reports 


All RELFRPs (N = 75) 

3.24 

3.39 

All West FRPs (N = 12) 

3.57 

3.42 

An Analysis of Utah’s K-3 Reading Improvement 
Program 

3.00 

3.00 

Characteristics of Arizona [and California and California 
(2008 Update)] School Districts in Improvement 

4.39 

3.33 

Course-taking Patterns and Preparation for 
Postsecondary Education in California’s Public University 
Systems Among Minority Youth 

2.17 

3.50 

Examining independent study high schools in California 

3.89 

4.11 

Examining the Links Between Grade 12 Mathematics 
Coursework and Mathematics Remediation in Nevada 
Public Colleges and Universities 

3.92 

3.33 

Implementation of the Weighted Student Formula Policy 
in San Francisco: a Descriptive Study of an Equity- 
Driven, Student-Based Planning and Budgeting Policy 

3.29 

3.00 

Measuring Resilience and Youth Development: The 
Psychometric Properties of the Healthy Kids Survey 

3.60 

3.33 

The Distribution of Teaching and Learning Resources in 
California’s Middle and High Schools 

4.12 

3.11 

The Reenrollment of High School Dropouts in a Large, 
Urban School District 

3.22 

3.44 

The Status of State-level Response to Intervention 
Policies and Procedures in the West Region States and 
Five Other States 

3.00 

3.33 

Training Early Intervention Assistants in California’s 
Community Colleges 

3.94 

3.78 

Trends in California Teacher Demand: A County and 
Regional Perspective 

4.29 

3.78 
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Table 13-2. Expert panel quality and relevance ratings for lES-published Fast Response Project 
reports and corresponding proposals from REL West and all 10 RELs combined (on 
a 5-point scale with 5 being the highest) (continued) 


Mean Ratings 

Quality 

Relevance 

Reports 


All REL FRPs (N = 91) 

3.81 

3.64 

All West FRPs (N = 15) 

4.14 

3.84 

A Multistate Review of Professional Teaching 
Standards(-) 

3.95 

3.50 

An Analysis of Utah’s K-3 Reading Improvement 
Program 

3.83 

3.58 

Characteristics of Arizona School Districts in 
Improvement 

4.44 

3.78 

Characteristics of California School Districts in 
Program Improvement 

4.44 

3.78 

Characteristics of California School Districts in 
Program Improvement: 2008 Update 

4.44 

3.78 

Course-taking Patterns and Preparation for 
Postsecondary Education in California’s Public 
University Systems Among Minority Youth 

4.20 

4.00 

Examining independent study high schools in 
California 

4.57 

4.17 

Examining the Links Between Grade 12 
Mathematics Coursework and Mathematics 
Remediation in Nevada Public Colleges and 
Universities 

3.86 

2.50 

Implementation of the Weighted Student Formula 
Policy in San Francisco: a Descriptive Study of an 
Equity-Driven, Student-Based Planning and 
Budgeting Policy 

3.71 

3.58 

Measuring Resilience and Youth Development: The 
Psychometric Properties of the Healthy Kids Survey 

4.18 

4.08 

The Distribution of Teaching and Learning Resources 
in California's Middle and High Schools 

4.05 

4.33 

The Reenrollment of High School Dropouts in a 
Large, Urban School District 

3.92 

4.17 

The Status of State-level Response to Intervention 
Policies and Procedures in the West Region States 
and Five Other States 

3.58 

3.50 

Training Early Intervention Assistants in California’s 
Community Colleges 

4.42 

4.50 

Trends in California Teacher Demand: A County and 
Regional Perspective 

4.48 

4.33 


Table Reads : For the 75 proposals for FRPs from all RELs, the mean quality dimension rating was 3.24. 


NOTE: N = Number of lES-published FRP reports released by December 1, 2009, or corresponding proposals, reviewed by the expert 
panels. (-) = A proposal was not available for the report, “A Multistate Review of Professional Teaching Standards.” 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 


Evaluation of the Regional Educational Laboratories: Interim Report 

138 


Figure 13-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from REL 
West 
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Figure Reads: 17.3 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong”) on 
a 5-point scale. 

NOTE: The distributions of ratings for proposal quality and relevance were based on 197 and 102 indicator-level ratings, respectively. The 
distributions of ratings for report quality and relevance were based on 330 and 172 indicator-level ratings, respectively. Ratings of “not 
applicable” were not included in these frequency distributions. 

N (Proposals) = 12 

N (Reports) = 15 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Fast Response Projects (For 
Proposals) (Appendix A). 
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Appendix A: Rubrics for Fast Response Project Expert 

Panel Review 


Expert panel members retrieved and scored their assigned FRP proposals and final reports through 
an online system. This appendix includes a paper version of the scoring mbrics, which mirror what 
was on the website. One mbric was used to rate proposals, and another was used to rate final 
reports. 
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RUBRIC AND SCORING FORM FOR THE QUALITY AND RELEVANCE OF 
REGIONAL EDUCATION LABORATORIES’ FAST RESPONSE PROJECTS (FOR FINAL REPORTS) 


REL: 

Project: 

Reviewer and Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: QUALITY OF THE PROJECT: Synthesis rating (mean of 1A through 1H) 


Indicator 1A. 

The research questions are clearly stated and are empirically testable. 


Indicator IB. 

The data sources are appropriate for addressing the research questions. 


Indicator 1C. 

The population being studied and the sampling techniques (i.e., sample size, sampling strategies) are appropriate for 
addressing the research questions. 


Indicator ID. 

Data collection instruments and their implementation are valid and reliable for the research questions. 


Indicator IE. 

The analyses conducted are appropriate for the data used. 


Indicator IF. 

All of the research questions are adequately addressed. 


Indicator 1G. 

The findings are clearly supported by the data. 


Indicator 1H. 

The limitations of the project are clearly described. 




DIMENSION 2: RELEVANCE OF THE PROJECT: Synthesis rating (mean of 2A through 2D) 


Indicator 2A. 

The report provides information that can be used to inform decisions about policies, programs, or practices. 


Indicator 2B. 

The report contributes new information to the topic being addressed. 


Indicator 2C. 

The report builds on key literature and/or previous research in the topic area. 


Indicator 2D. 

The report is easy to read and understand given the intended audience. 




GENERAL COMMENTS 
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SPECIFIC RATING SHEET 


Dimension 1: Quality ( Circle the number that best describes each indicator ) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

1A. The research questions are 
clearly stated and are 
empirically testable. 

The research 
questions are 
clearly articulated; 
key terms are well 
defined; and the 
research questions 
can be empirically 
tested. 


The research 
questions are 
stated, but could be 
clearer; not all key 
terms are defined; 
or the research 
questions are not 
stated in a way that 
easily lends to 
empirical testing. 


The report does not 
explicitly state any 
research questions; 
key terms are not 
defined; or stated 
questions cannot be 
empirically tested. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

IB. The data sources are 

appropriate for addressing the 
research questions. 

The study uses data 
sources that are 
most appropriate 
for addressing the 
research questions. 


The data sources 
used are adequate 
for addressing the 
research questions, 
but better sources 
are available. 


The data sources 
are either not 
identified explicitly 
or not well suited 
for addressing the 
research questions. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

1C. The population being studied 
and the sampling techniques 
(i.e., sample size, sampling 
strategies) are appropriate for 
addressing the research 
questions. 

The population 
being studied, 
sample size, and 
sampling strategies 
are most 
appropriate for 
addressing the 
research questions. 
Any potential bias 
has been addressed. 


The population 
being studied, 
sample size, and 
sampling strategies 
are reasonable for 
addressing the 
research questions, 
although there may 
be some potential 
sampling bias or 
minor concerns 
about sample size. 


The population 
being studied is not 
appropriate; the 
sample size is too 
small to address the 
research questions; 
and the sample 
may contain 
serious sampling 
bias; or population/ 
sampling was 
applicable to the 
study but was not 
addressed at all. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

ID. Data collection instruments 
and their implementation are 
valid and reliable for the 
research questions. 

All of the 
instruments are 
valid and reliable 
for addressing the 
research questions, 
and the data are 
collected using 
sound practices. 


Some but not all of 
the instruments are 
valid and reliable; 
the validity and/or 
reliability of some 
instruments is in 
question; or there 
are some concerns 
with the soundness 
of the data 
collection. 


The instruments 
have little if any 
validity or 
reliability for 
addressing the 
research questions, 
and there are 
serious problems 
with data 
collection, or 
instrumentation 
was applicable to 
the study but was 
not addressed at 
all. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

IE. The analyses conducted are 
appropriate for the data 
collected. 

The study uses the 
best and 
appropriate 
analysis methods 
for the research 
questions. 


The analysis 
methods are 
reasonable, 
although better 
ones could be 
applied within the 
context. 


The study uses few 
if any appropriate 
methods to analyze 
the data; or the 
analytic methods 
are not addressed. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

IF. All of the research questions 
are adequately addressed. 

All of the research 
questions are 
adequately 
addressed. 


Some but not all of 
the research 
questions are 
adequately 
addressed. 


Few if any of the 
research questions 
are adequately 
addressed. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

1G. The findings are clearly 
supported by the data. 

All of the findings 
are clearly 
supported by the 
data. 


Some but not all of 
the findings are 
supported by the 
data. 


Few if any of the 
findings are 
supported by the 
data. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

1H. The limitations of the project 
are clearly described. 

The limitations of 
the study are 
described clearly 
and 

comprehensively. 


The limitations of 
the study are 
acknowledged but 
may not be clear or 
comprehensive. 


The report includes 
little if any 
information on the 
limitations of the 
study. 



Comments on rating, required for NA: 
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Dimension 2: Relevance ( Circle the number that best describes each indicator ) 


Indicator 

Highly Relevant 

Relevant 

Adequate 

Marginally 

Relevant 

Not Relevant 

Not Applicable 

5 

4 

3 

2 

1 

NA 

2A. The report provides 

information that can be used 
to inform decisions about 
policies, programs, or 
practices. 

The report 

addresses 

important 

questions and 

provides 

exceptionally 

practical 

information and/or 
resources that can 
be used to inform 
decisions about 
policies, programs, 
or practices. 


The report 

addresses 

moderately 

important 

questions and 

provides some 

practical 

information and/or 
resources that can 
be used to inform 
decisions about 
policies, programs, 
or practices. 


The report 
addresses questions 
that are not 
important and 
provides little if 
any practical 
information or 
resources that can 
be used to inform 
decisions about 
policies, programs, 
or practices. 



Comments on rating, required for NA: 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly Relevant 

Relevant 

Adequate 

Marginally 

Relevant 

Not Relevant 

Not Applicable 

5 

4 

3 

2 

1 

NA 

2B. The report contributes new 
information to the topic being 
addressed. 

The information 
presented in the 
report is current, 
new, or makes a 
contribution to the 
intended audience 
in the region. 


The report is 
somewhat limited 
in providing 
current or new 
information or in 
making a 
contribution to the 
intended audience 
in the region. 


The report does not 
provide any current 
or new information 
or does not make a 
contribution to the 
intended audience 
in the region. 



Comments on rating, required for NA: 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly Relevant 

Relevant 

Adequate 

Marginally 

Relevant 

Not Relevant 

Not Applicable 

5 

4 

3 

2 

1 

NA 

2C. The report builds on key 
literature and/or previous 
research in the topic area. 

The study builds on 
a literature review 
of relevant, 
important, 
unbiased, and well- 
conducted 
research. 


The literature 
review is adequate, 
but may overlook 
some important 
work or include 
studies that are not 
well conducted, 
without 

consideration of 
their limitations. 


The literature 
review misses a 
large amount of 
key literature or is 
seriously biased. 



Comments on rating, required for NA: 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly Relevant 

Relevant 

Adequate 

Marginally 

Relevant 

Not Relevant 

Not Applicable 

5 

4 

3 

2 

1 

NA 

2D. The report is easy to read and 
understand given the intended 
audience. 

The report is well 
written throughout 
and can be easily 
understood by the 
audience identified 
in the report or by 
policymakers or 
practitioners at the 
state, district, or 
school level. 


Some but not all 
sections of the 
report are well 
written, and/or the 
report is too 
technical for the 
audience identified 
in the report or by 
policymakers or 
practitioners at the 
state, district, or 
school level. 


The report is 
poorly written 
throughout. 



Comments on rating, required for NA: 
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RUBRIC AND SCORING FORM FOR THE QUALITY AND RELEVANCE OF 
REGIONAL EDUCATION LABORATORIES’ FAST RESPONSE PROJECTS (FOR PROPOSALS) 


REL: 

Project: 

Reviewer and Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: QUALITY OF THE PROJECT: Synthesis rating (mean of 1A through IF) 


Indicator 1A. The proposed research questions are clearly stated and are empirically testable. 


Indicator IB. The proposed data sources are appropriate for addressing the research questions. 


Indicator 1C. The proposed population to be studied and the proposed sampling techniques (i.e., sample size, sampling strategies) are 
appropriate for addressing the research questions. 


Indicator ID. The proposed data collection instruments and their implementation are valid and reliable for the research questions. 


Indicator IE. The proposed analyses are appropriate for the proposed data collection. 


Indicator IF. All of the research questions can be adequately addressed. 




DIMENSION 2: RELEVANCE OF THE PROJECT: Synthesis rating (mean of 2A through 2C) 


Indicator 2A. The report will provide information that can potentially be used to inform decisions about policies, programs, or practices. 


Indicator 2B. The report will contribute new information to the topic area. 


Indicator 2C. The proposed project builds on key literature and previous research in the topic area. 




GENERAL COMMENTS 
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SPECIFIC RATING SHEET 


Dimension 1: Quality ( Circle the number that best describes each indicator ) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

1A. The proposed research 

questions are clearly stated 
and are empirically testable. 

The research 
questions are 
clearly articulated; 
key terms are well 
defined; and the 
research questions 
can be empirically 
tested. 


The research 
questions are 
stated, but could be 
clearer; not all key 
terms are defined; 
or the research 
questions are not 
stated in a way that 
easily lends to 
empirical testing. 


The proposal does 
not explicitly state 
any research 
questions; key 
terms are not 
defined, or stated 
questions cannot be 
empirically tested. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

IB. The proposed data sources 
are appropriate for addressing 
the research questions. 

The study will use 
data sources that 
are most 
appropriate for 
addressing the 
research questions. 


The data sources 
are adequate for 
addressing the 
research questions, 
but better sources 
are available. 


The data sources 
are either not 
identified explicitly 
or not well suited 
for addressing the 
research questions. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

1C. The proposed population to 
be studied and the proposed 
sampling techniques (i.e., 
sample size, sampling 
strategies) are appropriate for 
addressing the research 
questions. 

The proposed 
population to be 
studied, sample 
size, and sampling 
strategies are most 
appropriate for 
addressing the 
research questions. 


The proposed 
population to be 
studied, sample 
size, and sampling 
strategies are 
reasonable for 
addressing the 
research questions, 
although there may 
be some potential 
sampling bias or 
minor concerns 
about sample size. 


The proposed 
population to be 
studied is not 
appropriate; the 
sample size is too 
small to address 
the research 
questions; and the 
sample may 
contain serious 
sampling bias; OR 
population/ 
sampling was 
applicable to the 
study but was not 
addressed at all. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

ID. The proposed data collection 
instruments and their 
implementation are valid and 
reliable for the research 
questions. 

All of the proposed 
instruments are 
valid and reliable 
for addressing the 
research questions, 
and data will be 
collected using 
sound practices. 


Some but not all of 
the proposed 
instruments are 
valid and reliable; 
the validity and/or 
reliability of some 
instruments is in 
question; or there 
are some concerns 
with the soundness 
of the data 
collection. 


The proposed 
instruments have 
little if any validity 
or reliability for 
addressing the 
research questions, 
and there are 
serious problems 
with data 
collection, or 
instrumentation 
was applicable to 
the study but was 
not addressed at 
all. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

IE. The proposed analyses are 
appropriate for the proposed 
data collection. 

The study will use 
the best and 
appropriate 
analysis methods 
for the proposed 
data collection. 


The proposed 
analysis methods 
are reasonable, 
although better 
ones could be 
applied within the 
context. 


The study will use 
few if any 
appropriate 
methods to analyze 
the data, or the 
proposed analytic 
methods are not 
addressed. 



Comments on rating, required for NA: 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very Strong 

Strong 

Adequate 

Weak 

Very Weak 

Not Applicable 

5 

4 

3 

2 

1 

NA 

IF. All of the research questions 
can be adequately addressed. 

All of the proposed 
research questions 
can be adequately 
addressed. 


Some but not all of 
the proposed 
research questions 
can be adequately 
addressed. 


Few if any of the 
proposed research 
questions can be 
adequately 
addressed. 



Comments on rating, required for NA: 
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Dimension 2: Relevance ( Circle the number that best describes each indicator) 


Indicator 

Highly Relevant 

Relevant 

Adequate 

Marginally 

Relevant 

Not Relevant 

Not Applicable 

5 

4 

3 

2 

1 

NA 

2A. The report will provide 
information that can 
potentially be used to inform 
decisions about policies, 
programs, or practices. 

The report will 
address important 
questions and 
provide 
exceptionally 
practical 

information and/or 
resources that can 
potentially be used 
to inform decisions 
about policies, 
programs, or 
practices. 


The report will 
address moderately 
important 
questions and 
provide some 
practical 

information and/or 
resources that can 
potentially be used 
to inform decisions 
about policies, 
programs, or 
practices. 


The report will 
address questions 
that are not 
important and/or 
provide little if any 
practical 
information or 
resources that can 
potentially be used 
to inform decisions 
about policies, 
programs, or 
practices. 



Comments on rating, required for NA: 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly Relevant 

Relevant 

Adequate 

Marginally 

Relevant 

Not Relevant 

Not Applicable 

5 

4 

3 

2 

1 

NA 

2B. The report will contribute 
new information to the topic 
area. 

The information to 
be presented in the 
report will be 
current or new or 
make a 

contribution to the 
intended audience 
in the region. 


The information to 
be presented in the 
report will be 
somewhat limited 
in providing 
current or new 
information or in 
making a 
contribution to the 
intended audience 
in the region. 


The report will not 
provide any current 
or new information 
or will not make a 
contribution to the 
intended audience 
in the region. 



Comments on rating, required for NA: 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly Relevant 

Relevant 

Adequate 

Marginally 

Relevant 

Not Relevant 

Not Applicable 

5 

4 

3 

2 

1 

NA 

2C. The proposed project builds 
on key literature and previous 
research in the topic area. 

The proposed 
research builds on 
a literature review 
of relevant, 
important, 
unbiased, and well- 
conducted 
research. 


The literature 
review is adequate, 
but may overlook 
some important 
work or include 
studies that are not 
well conducted 
without 

consideration of 
their limitations. 


The literature 
review misses a 
large amount of 
key literature or is 
seriously biased. 



Comments on rating, required for NA: 


Appendix B: Inter-rater Agreement 


When protocols require multiple sets of raters to assign ratings to targets (in this case, proposals and 
final reports), it is important to assess two related but conceptually distinct concepts: interrater 
agreement and interrater reliability. Levels of interrater agreement (IRA) index the extent to which 
raters assign the same scores to common targets, while interrater reliability (IRR) indicates the extent 
to which raters provide consistency in relative judgments. For example, if one rater provides ratings 
that are exactly 2 points higher than another rater, measures of interrater reliability will be high, 
while interrater agreement will be lower. For the expert panel review of FRPs, IRA was identified as 
more conceptually appropriate for assessing the extent to which raters provided similar ratings. 

There are a wide variety of methods for assessing IRA and IRR, including measures such as Cohen’s 
kappa and various intraclass correlation coefficients (ICC). However, these measures were less than 
optimal for purposes of the current study. For example, Cohen’s kappa is a classic measure of 
agreement, but requires additional modifications (e.g., weighting schemes) to handle ranked data and 
is primarily designed around assessing agreement dyadically — e.g., between pairs of individual raters 
on individual items. The various ICC coefficients (c.f, Shrout and Fleiss 1979) are widely used to 
assess the consistency of ratings, but provide measures based on both IRA and IRR, rather than 
IRA alone (LeBreton and Senter 2008), do not provide a measure of agreement for individual 
products and are based on mean dimension-level ratings, rather than item-level responses. 

For the current report, IRA was indexed using the R wg( - measure (James, Demaree, and Wolf 1984). 
This measure is widely used in the psychometric, management, and psychology literatures; is 
applicable to multi-item ordinal response scales; and provides a measure of IRA for each proposal 
and final report being rated (as opposed to a single measure of IRA for the entire study) based on a 
definite number of raters. R o is given by the following formula: 



where J — the number of items in the rating scale, s x } is the mean of the observed variances for the J 
observed items, and erj is the expected variance when there is a complete lack of agreement among 
raters (e.g., where raters respond randomly to items). For the current report, cj =2, based on the 
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expected variance of a uniform response distribution for a 5-item response scale. R m ranges from 0 
to 1 , with 0 indicating a complete lack of agreement and 1 indicating perfect agreement. 

The analysis was conducted using the rwg.j function from the multilevel package in R version 2.10.1 
(R Foundation for Statistical Computing 2010); codes are also provided for SPSS in LeBreton and 
Senter (2008). Because the Rwg(j) formula expects numeric ratings, “NA” ratings were treated as 
missing data for purposes of the Rwg(j) analysis. 1 Missing data are known to have an effect on R ~ 
estimates (Newman & Sin 2009), such that responses missing not at random (MNAR) tend to 
produce overestimates of R^ ffl , with the level of bias dependent on the level of missingness in the 
data and the magnitude of the relationship between missingness and the variable of interest. 

In the current data set, only a small proportion of ratings involved an NA response (2.7 percent 
overall). The largest percentage of NA ratings were for final report ratings of relevance (4.3 percent). 
Moreover, the relationship between average ratings for a given product type (i.e., final report, 
proposal) and rating dimension (i.e., quality, relevance) were small (r = -0.01 for proposal quality, r — 
0.1 5 for proposal relevance, r — 0.20 for final report quality, and r — -0.03 for final report relevance). 
Together, these factors suggest that any bias which might result from treating NA responses as 
missing data should be small (e.g., less than 3 percent; see Table 1 and Figure 6 in Newman and Sin 
2009 for more detail). 

As with many metrics of IRA, what constitutes a “high” or “low” level of agreement will ultimately 
depend on the purposes of the ratings, but a review by LeBreton & Senter (2008) suggested the 
following interpretations of R^ ffl : ranges from 0.00 to 0.30 represent a lack of agreement; 0.31 to 
0.50 represents weak agreements; 0.51 to 0.70 is moderate agreement; 0.71 to 0.90 is strong 
agreement; and 0.91 to 1.00 is very strong agreement. In order to identify products for further 
reconciliation and re-review by raters, products were selected for reconciliation if they had an R wg , g 
of 0.30 or less on either quality or relevance (the two dimensions). 

In addition, for each product thus identified, raters were provided with a list of specific indicators 
for which the greatest level of disagreement existed, defined by the inter-rater variance for specific 
indicators. These were identified by selecting indicators for which ratings had a variance of two or 
more; this threshold was selected based on the expected variance of a uniform distribution of an 
ordinal scale with five response categories, such that indicators with variances less than two tended 


1 The base rwg.j function provided by the multilevel package had to be modified slightly to handle missing data; this is due to the 
default behavior of certain R functions for the mean and variance, rather than due to issues with the algorithm implemented by the 
rwg.j function or the Rwg(j) formula itself. 
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to have greater levels of agreement than might be expected based on chance responding. However, 
note that this threshold does not represent a statistical test of inter-rater agreement; rather, it was a 
heuristic designed to help raters quickly identify and resolve the largest discrepancies in their ratings 
in the limited time available. 
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Appendix C: REL Interview Protocol 


Read to REL Directors 

Thank you very much for taking the time to speak with us about your REL’s work. The 
purpose of the interview is to gain a better understanding of REL projects and activities. In 
preparation for this interview, we gathered background information on your REL’s work by 
reviewing the NLN website, your REL’s website, and your REL’s Updated Annual Plans. These 
sources were very informative. However, we recognize that speaking with you will provide us 
with a more comprehensive picture of your REL’s work. We are particularly interested in 
gaining infonnation on what you see as the major endeavors that your REL is presently working 
on or has already completed as they relate to the 10 REL missions identified in the Education 
and Sciences Reform Act 2002 (ESRA). Since ESRA refers to REL projects and activities as 
they relate to the 10 missions, this interview is structured utilizing the ESRA terminology, but 
we understand that RELs and IES typically do not use this language and often classify REL work 
by other designations (e.g., Bridge Events, Fast Response Projects) or by task number. 

For the purpose of this interview, major projects and activities are defined as up to the 
three most significant activities that your REL is working on or has completed for each of the 10 
missions. Please be sure that your responses to the interview questions are limited to the current 
REL contract and up to the three most significant projects or activities that your REL is working 
on or has completed for each of the missions. A major project or activity may fall under more 
than one mission. If this is the case, please explain in the interview how the project(s) or 
activity(s) encompasses more than one mission. Importantly, if you think that your REL has not 
completed a project or activity for each of the 10 missions, that is fine. Please just say so during 
the interview. 

Mission #1: Training or Technical Assistance 

1 . What are the major training or technical assistance projects or activities that your 
REL has completed or is in the process of completing? (Note: It is quite likely that 
Directors will refer to work completed under Task 1.1.) 

la. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., symposia, on-line 
workshop/webinar, conference, etc.); 

■ the rationale for implementing the project or activity; 
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the intended audience for the project or activity (Note: If it is not 
addressed in the above response, ask if the actual users are (or were) 
different from the intended audience. If there is (or was) a difference, 
ask for an explanation); and 

■ the current status of the project or activity (i.e., completed or in the 
process of completing). 

Mission #2: Dissemination 

2. What are the major dissemination activities that your REL completed or is in the 
process of completing? (Say: We are interested in dissemination activities that relate 
to scientifically valid research, information, reports, and publications.) (Note: It is 
quite likely that Directors will refer to work completed under Task 4.) 

2a. Please describe in detail each major activity, such as: 

■ the type of activity (i.e., report, other publication, symposia, on-line 
workshop/webinar, conference, etc.); 

■ the form of dissemination for each activity (i.e., email, journal, 
website, webinar, conference, etc.); 

■ the intended audience for the activity (Note: If it is not addressed in 
the above response, ask if the actual users are (or were) different from 
the intended audience. If there is (or was) a difference, ask for an 
explanation); and 

■ the current status of the activity (i.e., completed or in the process of 
completing). 

Mission #3: Needs Assessment or Needs Sensing 

3. What are the major needs assessment or needs sensing projects or activities that your 
REL has completed or is in the process of completing? (Note: It is quite likely that 
Directors will refer to work completed under Task 1 . 1 .2.) 

3a. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., email, phone call, survey, meeting, 
etc.); 

■ the rationale for implementing the project or activity; 

■ the intended audience for the project or activity (Note: If it is not 
addressed in the above response, ask if the actual users are (or were) 
different from the intended audience. If there is (or was) a difference, 
ask for an explanation); and 
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■ the current status of the project or activity (i.e., completed or in the 
process of completing). 

3b. Once you identified the respective needs from the projects or activities that 
you just mentioned, how did you ensure that your REL met each need (i.e., via 
report, symposium, workshop, conference, etc.)? 

Mission #4: Carrying Out Applied Research Projects 

4. What are the major applied research projects that your REL has completed or is in the 
process of completing to meet the needs of your region? Please note that this mission 
emphasizes original REL research . (Note: It is quite likely that Directors will refer to 
work completed under Task 2.) 

4a. Please describe in detail each major project, such as: 

■ the type of project (i.e., Randomized Controlled Trial); 

■ the rationale for implementing the project; 

■ the intended audience for the project (Note: If it is not addressed in the 
above response, ask if the actual users are (or were) different from the 
intended audience. If there is (or was) a difference, ask for an 
explanation); and 

■ the current status of the project (i.e., completed or in the process of 
completing). 

Mission #5: Providing Applied Research Projects in a Usable Format 

5. What are the major applied research projects that your REL has provided (or is in the 
process of providing) in usable formats to support the educational development 
activities and needs of your region? Please note that this mission emphasizes taking 
findings from existing research, including your REL’s work, and making them user- 
friendly for customers . (Note 1 : It is likely that Directors will say that Mission #4 and 
Mission #5 are the same. If this is the case, say that we recognize these missions are 
similar. However, since they are separate missions in ESRA, we still wanted to give 
the RELs the opportunity to respond. Also, repeat the line underlined above and say 
again that Mission #4 emphasizes original REL research.) (Note 2: It is quite likely 
that Directors will refer to work completed under Task 1.2.) 

5a. Please describe in detail each major project, such as: 

■ the type of project (i.e., Fast Response Project, symposia, on-line 
workshop/webinar, conference, etc.); 

■ the rationale for implementing the project; 
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■ the intended audience for the project (Note: If it is not addressed in the 
above response, ask if the actual users are (or were) different from the 
intended audience. If there is (or was) a difference, ask for an 
explanation); and 

■ the current status of the project (i.e., completed or in the process of 
completing). 

Mission #6: Coordination or Collaboration 

6. What are the major coordination or collaboration projects or activities that your REL 
has completed or is in the process of completing? (Note: Coordination or 
collaboration may take place with other RELs, other Department of Education- funded 
organizations, organizations not funded by the Department of Education, or experts, 
etc.) 

6a. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., email, phone call, survey, symposia, 
on-line workshop/webinar, conference, etc.); 

■ the name of the collaborative organization (i.e., Comprehensive 
Centers); and 

■ the current status of the project or activity (i.e., completed or in the 
process of completing). 

Mission #7: School Finance Systems 

7. What major activities or projects has your REL completed or is in the process of 
completing on school finance systems? 

7a. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., publication, symposia, on-line 
workshop/webinar, conference, etc.); 

■ the rationale for implementing the project or activity; 

■ the intended audience for the project or activity (Note: If it is not 
addressed in the above response, ask if the actual users are (or were) 
different from the intended audience. If there is (or was) a difference, 
ask for an explanation.); and 

■ the current status of the project or activity (i.e., completed or in the 
process of completing). 
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Mission #8: Alternative Administrative Structures 


8. What major activities or projects has your REL completed or is in the process of 
completing on alternative administrative structures? 

8a. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., publication, symposia, on-line 
workshop/webinar, conference, etc.); 

■ the rationale for implementing the project or activity; 

■ the intended audience for the project or activity (Note: If it is not 
addressed in the above response, ask if the actual users are (or were) 
different from the intended audience. If there is (or was) a difference, 
ask for an explanation.); and 

■ the current status of the project or activity (i.e., completed or in the 
process of completing). 

Mission #9: School Improvement Plans 

9. What major activities or projects has your REL completed or is in the process of 
completing that involved/involves bringing experts together to develop and 
implement school improvement plans? 

9a. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., publication, symposia, on-line 
workshop/webinar, conference, etc.); 

■ the rationale for implementing the project or activity; 

■ the intended audience for the project or activity (Note: If it is not 
addressed in the above response, ask if the actual users are (or were) 
different from the intended audience. If there is (or was) a difference, 
ask for an explanation.); and 

■ the current status of the project or activity (i.e., completed or in the 
process of completing). 

Mission #10: Technology in Education 

10. What major activities or projects has your REL completed or is in the process of 
completing on developing innovative approaches to the application of technology in 
education? 

10a. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., publication, symposia, on-line 
workshop/webinar, conference, etc.); 
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■ the rationale for implementing the project or activity; 

■ the intended audience for the project or activity (Note: If it is not 
addressed in the above response, ask if the actual users are (or were) 
different from the intended audience. If there is (or was) a difference, 
ask for an explanation.); and 

■ the current status of the project or activity (i.e., completed or in the 
process of completing). 


Other 

11. What other major activities or projects has your REL completed or is in the process of 
completing? 

11a. Please describe in detail each major project or activity, such as: 

■ the type of project or activity (i.e., publication, symposia, on-line 
workshop/webinar, conference, etc.); 

■ the rationale for implementing the project or activity; 

■ the intended audience for the project or activity (Note: If it is not 
addressed in the above response, ask if the actual users are (or were) 
different from the intended audience. If there is (or was) a difference, 
ask for an explanation.); and 

■ the current status of the project or activity (i.e., completed or in the 
process of completing). 

Additional Comments 

12. In gathering background information on your REL’s work, we have come to 
understand that there are several fairly standard projects or activities that most RELs 
complete. We plan to ask you for your thoughts on several of these projects or 
activities (i.e., Ask-A-REL). 

Part II: Follow-up Questions 

Read to REL Directors: 

The second part of this interview focuses on specific projects or activities that we 
have learned about by reading REL web pages, Updated Annual Plans, and the NLN website. 

We are interested in learning whether your REL participated in these specific projects or 
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activities. Further, if your REL did participate in any of these projects or activities, we would 
like to know more about the specifics of your REL’s participation. 

1 . As you know, we have a list of the Bridge Events that your REL has held in the past, 
as well as those that your REL plans to hold in the future. Would you like to share 
with us any additional information concerning your REL’s Bridge Events that you 
thi nk is important for us to know? (Note: We will share the list of Bridge Events with 
the REL Directors prior to the interview.) 

(Regardless of the response to the question above, ask:) 

la. Thinking back to Part I of the interview, under what mission would you 
classify the majority of your REL’s Bridge Events? (Note: Show a list of the 
missions to the REL Director.) 

2. Does your REL currently participate or in the past has your REL participated in 
Thursdays @3? 

(If the response is yes, then ask:) 

2a. Approximately, how often does your REL participate in Thursdays @ 3? 

2b. Please describe the purpose of Thursdays @ 3. 

2c. What is your REL’s role at Thursdays @ 3? 

2d. Thinking back to Part I of the interview, under what mission would you 
classify Thursdays @ 3? (Note: Show a list of the missions to the REL Director.) 

3. Does your REL currently participate or in the past has your REL participated in REL 
Director Meetings? 

(If the response is yes, then ask:) 

3a. Approximately how often does your REL participate in the meetings? 

3b. Please describe the purpose of the meetings. 

3c. What is your REL’s role at the meetings? 

3d. Thinking back to Part I of the interview, under what mission would you 
classify the meetings? (Note: Show a list of the missions to the REL Director.) 
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4. Does your REL currently participate or in the past has your REL participated in the 
REL Director Webcast Series? 

(If the response is yes, then ask:) 

4a. Approximately how often does your REL participate in these series? 

4b. Please describe the purpose of the series. 

4c. What is your REL’s role in the series? 

4d. What, if any, relationship do the REL Director Meetings have with the REL 

Director Webcast Series? 

4e. Thinking back to Part I of the interview, under what mission would you 

classify the series? (Note: Show a list of the missions to the REL Director.) 

5. Does your REL currently participate or in the past has your REL participated in any 
of the following working groups, as well as additional working groups: ARRA, Data, 
Dissemination, Rural, or External Technical? 

(If the response is yes, then ask:) 

5 a. What are the names of the working groups that your REL participated in? 

5b. Please describe each working group, such as: 

■ an approximation of how often you participate(d) in the working 
group; 

■ the purpose of the working group; and 

■ your REL’s role in the working group. 

5c. Thinking back to Part I of the interview, under what mission would you 

classify working groups? (Note: Show a list of the missions to the REL Director.) 

6. As we noted earlier in the interview, to better understand the work of your REL, we 
visited your REL-related website. Please talk about how you thi nk about this website. 
For example, what role does the website play? What is its purpose (i.e., 
dissemination, Ask-A-REL, needs assessment, etc.)? (Note: It is important that the 
interviewer review this website prior to the interview, as the websites are highly 
variable.) 
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(If Ask-A-REL is not mentioned, then ask:) 

6a. In reviewing the website, we noticed that visitors are able to ask your REL 
questions via the feature, “Ask-A-REL.” Please talk about the role of Ask-A- 
REL. 

6b. Who is the website’s intended audience? 

6c. How often is the website updated with new information (i.e., FRPs, Bridge 
Events, etc.)? (Note: Some websites DO NOT have FRPs and Bridge Events 
listed. Rather, they are very general.) 

6d. Thinking back to Part I of the interview, under what mission would you 
classify the website? (Note: Show a list of the missions to the REL Director.) 

7. Does your REL currently participate or in the past has your REL participated in the 
REL Reference Desk? 

(If the response is yes, then ask:) 

7a. Please describe the REL Reference Desk, such as: 

■ the purpose of the Reference Desk; 

■ the Reference Desk’s intended audience; and 

■ your REL’s specific involvement with the Reference Desk. 

7b. Is there a relationship between the Reference Desk and Ask-A-REL? If so, 
please explain. 

7c. Thinking back to Part I of the interview, under what mission would you 
classify the Reference Desk? (Note: Show a list of the missions to the REL 
Director.) 

8. Does your REL currently participate or in the past has your REL participated in the 
REL Request and Response Database? 

(If the response is yes, then ask:) 

8a. Please describe the REL Request and Response Database, such as: 

■ the purpose of the database; 

■ the database’s intended audience; and 

■ your REL’s specific involvement with the database. 

8b. Thinking back to Part I of the interview, under what mission would you 
classify the Request and Response Database? (Note: Show a list of the missions to 
the REL Director.) 
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9. Please tell us about the role of your Governing Board (i.e., its purpose). 

9a. How often does the Governing Board meet? 

9c. Thinking back to Part I of the interview, under what mission would you 
classify the Governing Board? (Note: Show a list of the missions to the REL 
Director.) 

10. We would like to know, from your perspective, the role of the NLN website. 

More specifically: 

10a. What is your REL’s involvement with the website? 

10b. What are the various functions of the website? For example, is the website 
used as a means for your REL to communicate with IES, for IES to communicate 
with your REL, and/or for your REL to communicate with other RELs? 

10c. How frequently does your REL use the website? (Specific to Mid- Atlantic: 
We understand that your REL has the responsibility of maintaining the website. 
What does this responsibility entail? How often do you update the website with 
new material?) 

lOd. Thinking back to Part I of the interview, under what mission would you 
classify the NLN website? (Note: Show a list of the missions to the REL 
Director.) 

11. We are interested in better understanding how your REL handles technical assistance 
requests. 

Specifically: 

11a. Does your REL work with outside entities or individuals to respond to 
technical assistance requests? If so, what types (i.e., content, scope, etc.) of 
requests does your REL ask other entities or individuals to respond to? 
Conversely, what types (i.e., content, scope, etc.) of technical assistance requests 
do other entities or individuals ask your REL to respond to? 
lib. What are the types (or names) of entities and individuals that your REL 
works with in responding to technical assistance requests? 
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12. Does your REL currently have or in the past have you had priority areas assigned to 
your REL? 

(If the response is yes, then ask:) 

12a. Please describe the rationale for assigning priority areas to the RELs. 

More specifically: 

■ How were the priority areas assigned? 

■ What priority areas was your REL given? 

■ Have priority areas remained the same over time or have they 
changed? 

o If they have changed, how so? 

■ How does your REL ensure that customers know your REL’s priority 
areas? 

■ Does the assignment of priority areas among the RELs serve as a 
means of coordination? 

o If so, how? 
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List of Missions 


Mission #1: Training or Technical Assistance 

Mission #2: Dissemination 

Mission #3: Needs Assessment or Needs Sensing 

Mission #4: Carrying Out Applied Research Projects 

Mission #5: Providing Applied Research Projects in a Usable Fonnat 

Mission #6: Coordination or Collaboration 

Mission #7: School Finance Systems 

Mission #8: Alternative Administrative Structures 

Mission #9: School Improvement Plans 

Mission #10: Technology in Education 

Other 
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Appendix D: Technical Working Group 

Members 


Expert Advisory Panel Members 


Judy Arter 

Educational Testing Service 
Rolf Blank 

Council of Chief State School Officers 
Gregg Jackson 

George Washington University 

Conrad Katzenmeyer 
University of Central Florida 

Larry Ludlow 
Boston College 

Larry Orr 

Independent Consultant 
Colleen Serement 

Maryland State Department of Education 
Deb Sigman 

California Department of Education 
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